# GrowthBook Docs > Open source feature flagging and A/B testing platform. This file contains all documentation content in a single document following the llmtxt.org standard. ## Audit logs import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; GrowthBook keeps an audit log of all actions taken on the platform. You can access this page from the Settings → Log in the left navigation. ![Audit log page](/images/using/audit-logs.png) These logs are useful for auditing what users have done, and figuring out the cause of any issues. These logs are not user editable and the log is exportable. --- ## User & Team Permissions import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; In the context of GrowthBook, a user, or member, is an individual who has access to your organization's GrowthBook account. Each user is assigned a role that determines the level of access they have within the application. GrowthBook offers a range of roles, from `No Access` to `Admin` and even custom roles, each with a specific set of permissions (see below). GrowthBook's user permissions system allows organizations to define the level of access each user has within the application. This granular control ensures that users can only access the resources they need to perform their job, enhancing security and privacy. :::note Plan availability On the Free plan, members can only be assigned the `Admin` role — assigning any other role (including custom roles) requires a Pro or Enterprise account. Organizations created before this limit was introduced are unaffected, and existing role assignments always keep working. ::: ## Adding Users Team members can be added to your GrowthBook account via the `Settings` → `Members` page. From this main page, you'll see an "Invite Member" button on the right. If you do not see this button, you do not have permission to add members to your organization. Clicking on "Invite Member" will open a modal window from which you can choose some options for the new user. ![Inviting members modal](/images/using/invite-member-modal.png) Each GrowthBook user needs an email address, and you can select what global permissions you want to assign (See below for a full list of permission levels). Inviting a new member to your organization will send an email to that user inviting them to join. It is possible for a user to join more than one organization. If the user is a member to multiple organizations, they will see a drop-down next to their email address on the top right of the page, where they can select the organization they want to work in. ### Environment Specific Limits ![Environment Specific Limits](/images/user-permissions-env-specific.png) GrowthBook's user permissions also include environment specific limits. This permission level applies only to `Engineer` and `Experimenter` roles in Pro or Enterprise accounts. It allows you to limit the feature flags and experiments a user can manage to specific environments. For example, you can allow an `Engineer` to create and run experiments in a staging environment, but not in production. ### Project Specific Permissions Besides the global permissions, you can also assign project-specific permissions to users. This allows you to define a user's default role across all projects and select per-project overrides. For example, a new user could be a `collaborator` by default for all projects, but on the `mobile` project, they could be an `experimenter` so they can manage all feature flags and experiments assigned to that project. ### How permissions are evaluated GrowthBook first determines whether the action is scoped to a specific project. **Actions that target a project** (for example, editing a feature flag that belongs to a project) are evaluated using project-specific roles: - If a project-specific role applies (the user's own, or one from any team they're on), only project-specific roles are used for that project; the global role is ignored. The effective permissions are the union of those project-specific roles (the user gets a permission if any of them grants it). - Otherwise, the global role applies - itself the union of the user's global role and the global roles of any teams they're on. Because the global role is ignored whenever a project-specific role applies, a project-specific role can _reduce_ access below the global role. For example, a user whose global role is `Engineer` but who has a `Read Only` role on a project is read-only on that project. **Actions that are not scoped to a project** are evaluated against the global role. This includes organization-level settings and resources that always live at the org level, such as `Dimensions`, `Namespaces`, and `Presentations`. For organizations without a Pro or Enterprise account, permissions are evaluated solely against the global role. :::note Default role for self-registered users If your organization has enabled the setting to allow verified users to automatically join your organization, they will receive the `Collaborator` role by default when they join. However, you can change your organizations' default role at the bottom of the Team page. ::: ### Self-registering and Automatic Approvals If users create an account with a verified email address that matches the domain of your account owner, they will be presented with an option to join your organization. If you have not selected `Automatically approve new verified users` (which is the default), those users will be listed at the bottom of the page under a section called `Pending Members`. From this list, you'll have the option of approving or deleting these self-registered users. ![Self-registering and Automatic Approvals Toggle](/images/using/auto-approve-members.png) If you are the account owner, you will see a toggle at the top of the members' page that allows you to automatically approve new members who match your domain. This means that instead of being placed in your `pending members` list, they will automatically join your organization. ### Removing Users To remove a user from your organization, you can click on the three dots next to their name and select `Remove User`. This will remove the user from your organization and revoke their access to all projects and resources within your organization. ## Permissions Fine-tuning user permissions in an application like GrowthBook ensures a tailored experience, empowering organizations to grant precisely defined access levels. This granular control not only enhances security but also enables teams to collaborate efficiently while safeguarding sensitive features or data. Organizations using GrowthBook have a number of different ways of defining a user's permission level, depending on the organization's plan. Regardless of the plan, all organizations can assign a global role when inviting a user which defines their permissions across all projects. If you have a Pro or Enterprise account, you can also assign project-level roles for each user. For example, you can assign a user the global role of `Collaborator`, allowing them to view features and experiments, add comments, and contribute ideas. You can then assign them an `Experimenter` role for a specific project, which allows them to create and run experiments, but only for that project. And, for our Enterprise organizations, we offer the ability to create `Teams`, which are groups of users, all of which inherit the roles and permissions of the Team they're on. The table below lists the roles available in GrowthBook and their associated permissions. | | No Access | Read Only | Collaborator | Engineer | Analyst | Experimenter | Project Admin | Admin | | :-------------------------: | :-------: | :--------------: | :---------------------------: | :-------------------------------------------: | :---------------------------------------------------------------: | :---------------------------------------------------------------: | :---------------------------------------------------------------: | :---------------------------------------------------------------: | | Feature Flags | - | View | ViewComment | ViewCommentAddEdit | ViewComment | ViewCommentAddEdit | ViewCommentAddEdit | ViewCommentAddEdit | | Experiments | - | View | ViewComment | ViewCommentEdit | ViewCommentAddEditRun Queries | ViewCommentAddEditRun Queries | ViewCommentAddEditRun Queries | ViewCommentAddEditRun Queries | | Metrics | - | View | View | View | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Dimensions | - | View | View | View | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Segments | - | View | View | View | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Datasources | - | View | View | View | ViewEdit\* | ViewEdit\* | ViewEdit\* | ViewAddEdit | | Ideas | - | View | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | | SDK Connections | - | ViewAdd | ViewAdd | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Attributes | - | View | View | ViewAddEdit | View | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Namespaces | - | View | View | ViewAddEdit | View | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Environments | - | View | View | ViewAddEdit | View | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Saved Groups | - | View | View | ViewAddEdit | View | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Tags | - | - | - | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | ViewAddEdit | | Product Analytic Dashboards | - | View | View | View | ViewAddEdit | View | View | ViewAddEdit | | Slack Integration | - | - | - | - | - | - | - | ViewAddEdit | | Manage Projects | - | - | - | - | - | - | Yes | Yes | | Manage Team | - | - | - | - | - | - | - | Yes | | Manage Plan | - | - | - | - | - | - | - | Yes | | Manage Billing | - | - | - | - | - | - | - | Yes | \*_Limited to editing a subset of data source settings - identifier types, experiment assignment queries, Jupyter Notebook queries and Data Pipeline settings. Editing datasource name, projects, description, and connection parameters requires Admin permissions._ ### How does the No Access role work? In some cases, an organization might want to hide certain projects from a user entirely. This is possible with the `No Access` role. The `No Access` role can either be applied as the user's global role, or it can be a project-specific role. In the event you want a user to have access to a subset of projects, you can give them a global role of `No Access` and then give project-specific permissions for the projects you want them to be able to view. If, however, you only want to hide a certain project from a user, you can assign their project-level role of `No Access` which will make it so the user isn't able to view the project. If you need to apply these rules to many users at once, you can create a Team with the permissions needed, and then add users to that team. Keep in mind how project-level and global roles combine when a user is on one or more teams: - For a project where the user (or any team they are on) has an **explicit** project-level role, their access to that project is the union of those explicit project-level roles. Global roles are **not** used for that project. - For a project where no explicit project-level role applies, the user's global role (merged with their teams' global roles) is used instead. So to hide a specific project from a user, make sure no explicit project-level role grants them access to it: set their own project-level role for that project to `No Access`, and do the same for any team they are on that would otherwise grant access to it. ### How does the Project Admin role work? The Project Admin role provides full access to all features and experiments within a project, similar to the `Experimenter` role. Additionally, Project Admin allows you to manage project settings and change project roles for other members within that project. This makes it ideal for delegating project-level administration without granting full `Admin` permissions. The Project Admin role can be applied at either the global or project level. When applied globally, the user has Project Admin permissions across all projects. When applied at the project level, the user has Project Admin permissions only for that specific project. Key differences from the `Admin` role: - Project Admin can manage project settings and project member roles, but cannot manage organization-wide settings - Project Admin cannot add or remove users, manage user's global roles, add/remove users to teams, manage billing, or organization-level configurations :::note Organization-level resources are not project-specific Within GrowthBook, not all resources are project-specific. For example, `Dimensions`, `Namespaces`, and `Presentations` all live at the organization level. This means that all users, regardless of role, will be able to view these resources. If security of these resources is paramount to your organization, we recommend creating a separate organization, and keeping the resources you want to hide in that organization. ::: ### Teams Enterprise organizations using GrowthBook can create Teams with distinct capabilities. When setting up a Team, you have the option to define both a global role and project-level roles, much like how you do for individual users. Once a Team is established, multiple users can be added to it. Any user added to a Team will automatically inherit all permissions assigned to that Team. This feature becomes particularly useful when combined with [GrowthBook's SCIM integration](/integrations/scim), enabling automated user provisioning and de-provisioning. To create a Team, you can go to `Settings` → `Members` → `Team` via the Sidebar and then select the `Teams` tab at the top of the page. Here, you can create and configure various Teams, before adding members to a Team. When evaluating whether or not a user has permission to perform a certain action, we will merge the user's permissions with the permissions inherited from all the Teams the user is on. So if the user's global role is `Collaborator` but they're on a Team that grants them `Engineer` permissions, that user's permission will then be a merger of the `Collaborator` and `Engineer` roles. ### Custom roles Enterprise organizations using GrowthBook also have the added flexibility of defining custom roles, which enable organizations to fine-tune a role's permissions. These custom roles can be used just like a standard role and can be applied to users and teams at both the global and project levels. A custom role can also be set as your organization's default role, so if you have auto-join enabled, new members will automatically receive the organization's default role, even if it is a custom role. When creating a custom role, you can either create a role from scratch or duplicate an existing role and then update the role's description along with the policies, which control the role's permissions. Once created, the name of a custom role cannot be changed. If you need to change the name, you will need to duplicate the role and set the new name before saving. Once saved, you'll need to update users to use this new role. #### Policies & Permissions When creating and editing custom roles, organizations have the ability to select specific policies for each role, where the policy contains the underlying permissions. Below, we've outlined the current policies and their associated permissions. If your use case is not met with the current policies, please let us know by creating a [Github Issue](https://github.com/growthbook/growthbook/issues). | Policy Group | Policy | Description | Permissions | | ----------------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Global** | ReadData | View all resources - features, metrics, experiments, data sources, etc. | readData | | | Comments | Add comments to any resource | readData, addComments | | **Feature Flags, Configs, and Constants** | FlagsFullAccess | Perform every lifecycle action listed below for Feature Flags, Configs, and Constants | readData, createFeatures, createConfigs, createConstants, deleteFeatures, deleteConfigs, deleteConstants, editFeatureDrafts, editConfigDrafts, editConstantDrafts, reviewFeatures, reviewConfigs, reviewConstants, publishFeatures, publishConfigs, publishConstants, revertFeatures, revertConfigs, revertConstants | | | ↳ FlagsCreate | Create new Feature Flags, Configs, and Constants. Enabling a new Feature Flag in an environment also requires Publish access to that environment. | readData, createFeatures, createConfigs, createConstants | | | ↳ FlagsEditDrafts | Create and edit drafts, request review, and discard drafts without changing live behavior | readData, editFeatureDrafts, editConfigDrafts, editConstantDrafts | | | ↳ FlagsReview | Approve a draft or request changes | readData, reviewFeatures, reviewConfigs, reviewConstants | | | ↳ FlagsPublish | Change live behavior by publishing a draft, saving directly, unarchiving a resource, or toggling an environment. Direct saves also require Edit access. | readData, publishFeatures, publishConfigs, publishConstants | | | ↳ FlagsRevert | Restore a previously published revision | readData, revertFeatures, revertConfigs, revertConstants | | | ↳ FlagsDelete | Archive a live resource or permanently delete one that is already archived | readData, deleteFeatures, deleteConfigs, deleteConstants | | | FlagsBypassApprovals | Publish without an approval, force-publish an out-of-date draft, or unlock a Config. Validation and Custom Hooks still run unless the caller explicitly uses the corresponding REST override. | readData, bypassApprovalFeatures, bypassApprovalConfigs, bypassApprovalConstants | | | ArchetypesFullAccess | Create, edit, and delete saved User Archetypes for Feature Flag debugging | readData, manageArchetype | | **Saved Groups** | SavedGroupsFullAccess | Perform every lifecycle action listed below for Saved Groups | readData, createSavedGroups, deleteSavedGroups, editSavedGroupDrafts, reviewSavedGroups, publishSavedGroups, revertSavedGroups | | | ↳ SavedGroupsCreate | Create new Saved Groups | readData, createSavedGroups | | | ↳ SavedGroupsEditDrafts | Create and edit drafts, request review, and discard drafts without changing the live Saved Group | readData, editSavedGroupDrafts | | | ↳ SavedGroupsReview | Approve a draft or request changes | readData, reviewSavedGroups | | | ↳ SavedGroupsPublish | Publish a draft, save directly, or unarchive a Saved Group. Direct saves also require Edit access. | readData, publishSavedGroups | | | ↳ SavedGroupsRevert | Restore a previously published revision | readData, revertSavedGroups | | | ↳ SavedGroupsDelete | Archive a Saved Group or permanently delete one that is already archived | readData, deleteSavedGroups | | | SavedGroupsBypassApprovals | Publish without an approval or force-publish an out-of-date draft | readData, bypassApprovalSavedGroups | | | BypassSavedGroupSizeLimit | Exceed the organization's size limits for a Saved Group | readData, bypassSavedGroupSizeLimit | | **Experiments** | ExperimentsFullAccess | Create, edit, and delete experiments. Does not include Visual Editor access. | readData, createAnalyses, runQueries | | | ExperimentsPublish | Start and stop experiments, which changes what is sent to SDKs. | readData, runExperiments | | | VisualEditorFullAccess | Use the Visual Editor to implement experiment changes. | readData, manageVisualChanges | | | SuperDeleteReports | Delete custom reports made by other users. Typically assigned to admins only. | readData, superDeleteReport | | | TemplatesFullAccess | Create, edit, and delete experiment templates | readData, manageTemplates | | | HoldoutsFullAccess | Create, edit, and delete holdouts | readData, createAnalyses, runQueries | | **Metrics and Data** | DataSourcesFullAccess | Create, edit, and delete data sources | readData, createDatasources, editDatasourceSettings, runQueries | | | DataSourceConfiguration | Edit existing data source configuration settings (identifier types, experiment assignment queries) | readData, editDatasourceSettings, runQueries | | | RunQueries | Execute queries against data sources. Required to refresh experiment results. Does not include SQL Explorer access. | readData, runQueries | | | SqlExplorerFullAccess | Create, run, edit, and delete SQL Explorer queries | readData, runSqlExplorerQueries | | | MetricsFullAccess | Create, edit, and delete regular metrics (does not include Fact Metrics) | readData, createMetrics, runQueries, createMetricGroups | | | FactTablesFullAccess | Create, edit, and delete fact tables, metrics, and filters. | readData, manageFactTables, manageFactMetrics, manageFactFilters, runQueries | | | FactMetricsFullAccess | Create, edit, and delete fact metrics and filters. | readData, manageFactMetrics, manageFactFilters, runQueries | | | DimensionsFullAccess | Create, edit, and delete dimensions | readData, createDimensions, runQueries | | | SegmentsFullAccess | Create, edit, and delete segments | readData, createSegments, runQueries | | | ManageOfficialResources | Create, edit, and delete official resources such as Fact Tables, Metrics, and Segments. | readData, manageOfficialResources, runQueries | | **Management** | IdeasFullAccess | Create, edit, and delete ideas | readData, createIdeas | | | PresentationsFullAccess | Create, edit, and delete presentations | readData, createPresentations | | **Product Analytic Dashboards** | GeneralDashboardsFullAccess | Create, edit, and delete Product Analytics dashboards. | readData, manageGeneralDashboards | | **Session Replay** | SessionReplayViewAccess | View and play back recorded user sessions. | readData, viewSessionReplay | | | SessionReplayFullAccess | View, play back, and delete recorded user sessions (single, bulk, and DSR-driven deletions). | readData, viewSessionReplay, deleteSessionReplay | | **SDK Configuration** | SDKConnectionsFullAccess | Create, edit, and delete SDK Connections | readData, manageSDKConnections, manageSDKWebhooks | | | AttributesFullAccess | Create, edit, and delete targeting attributes | readData, manageTargetingAttributes | | | EnvironmentsFullAccess | Create, edit, and delete environments | readData, manageEnvironments | | | NamespacesFullAccess | Create, edit, and delete namespaces | readData, manageNamespaces | | **Settings** | GeneralSettingsFullAccess | Edit organization general settings | readData, organizationSettings | | | NorthStarMetricFullAccess | Configure North Star metrics | readData, manageNorthStarMetric | | | TeamManagementFullAccess | Invite users, delete users, change user roles, add/remove users from teams. | readData, manageTeam | | | CustomRolesFullAccess | Create, edit, and delete custom roles | readData, manageTeam, manageCustomRoles | | | CustomFieldsFullAccess | Create, edit, and delete custom fields | readData, manageCustomFields | | | ProjectsFullAccess | Create, edit, and delete projects and change project roles for other members. Can be applied at the global or project level. | readData, manageProjects, createProjects, deleteProjects | | | ProjectAdminAccess | Manage project settings and change project roles for other members. | readData, manageProjects | | | TagsFullAccess | Create, edit, and delete tags | readData, manageTags | | | APIKeysFullAccess | Create, edit, and delete API secret keys. Not required to create Personal Access Tokens. | readData, manageApiKeys | | | IntegrationsFullAccess | Set up and configure integrations - GitHub, Vercel, etc. | readData, manageIntegrations | | | EventWebhooksFullAccess | Create, edit, and delete event-based webhooks. Used for Slack/Discord notifications. | readData, manageEventWebhooks, viewAuditLog | | | BillingFullAccess | View and edit license key. View invoices and update billing info. | readData, manageBilling | | | AuditLogsFullAccess | View and export audit logs | readData, viewAuditLog | | | DecisionCriteriaFullAccess | Create, edit, and delete decision criteria, part of the experiment decision framework. | readData, manageDecisionCriteria | | | CustomHooksFullAccess | Create, edit, and delete Global, Project, and Config-scoped custom hooks. A hook scoped to a single Feature Flag instead takes edit rights on that flag (FlagsEditDrafts). | readData, manageCustomHooks | #### Choosing individual permissions You do not have to grant **Full access**. Expand that policy in the role editor to select only the actions a role needs. For example, a role with **Edit** but not **Publish** can prepare drafts but cannot change live behavior. Feature Flags, Configs, and Constants share the same set of policies. Create, Publish, Revert, and Archive & delete respect a role's environment restrictions. GrowthBook checks only the environments affected by the action. For example, archiving a Feature Flag checks the environments where that flag is currently enabled. Permanently deleting an already archived resource does not require access to any environment. Some changes are not environment-specific at all. A Constant's base value and a Config that is not scoped to particular environments apply everywhere rather than to any one environment, so there is no environment for GrowthBook to check. A role with Publish for the Project can publish them regardless of its environment restrictions. Moving a resource to another Project requires the relevant permission in both the current and destination Projects. #### Deactivating Roles As we do not support the ability for an organization to delete a standard role, we have introduced the ability for enterprise organizations to deactivate both standard and custom roles. When a role is deactivated, we remove the role from the roles dropdown when adding a new user or updating an existing user's role. If you deactivate a role that is assigned to a user, the user will experience no changes to their permission level. The deactivation of the role simply removes it from the role options. The only guardrail in place around deactivating roles is that you cannot deactivate your organization's default role. --- ## API # API GrowthBook offers a full REST API for interacting with the application. [View REST API Docs](/api/) ## SDK Connection Endpoints In addition to the REST API above, there is one additional readonly endpoint - the SDK Connection Endpoint. The SDK Connection Endpoint provides readonly access to a subset of your feature flag data, just enough for the [GrowthBook SDKs](/lib) to assign values to users. They are meant to be public and do not require authentication to view. In **GrowthBook Cloud**, the SDK Connection Endpoints are served from our global CDN: `https://cdn.growthbook.io/api/features/...`. If you are self-hosting, you can use [your own CDN](/self-host/cdn) or run the [GrowthBook Proxy server](/self-host/proxy), which provides built-in caching and performance optimizations. SDK Connection Endpoints are scoped to a single Environment (e.g. `dev` or `production`) and can also be scoped to a single Project. Manage all of your SDK Connections on the **Features → SDKs** page. Typescript Type Definition ```ts interface SDKEndpointResponse { status: 200; features: { [key: string]: FeatureDefinition } } interface FeatureDefinition { defaultValue: any; rules?: FeatureDefinitionRule[]; } interface FeatureDefinitionRule { force?: any; weights?: number[]; variations?: any[]; hashAttribute?: string; hashVersion?: number; seed?: string; namespace?: [string, number, number]; key?: string; coverage?: number; condition?: any; meta?: VariationMeta[]; name?: string; phase?: string; } interface VariationMeta = { passthrough?: boolean; key?: string; name?: string; } ``` Example JSON object ```json { "status": 200, "features": { "feature-key": { "defaultValue": true }, "another-feature": { "defaultValue": "blue", "rules": [ { "condition": { "browser": "firefox" }, "force": "green" } ] } } } ``` ### Encryption If you've enabled encryption for your SDK endpoint, the response format changes: Typescript Type Definition ```ts interface SDKEncryptedEndpointResponse { status: 200; encryptedFeatures: string; } ``` Example JSON object ```json { "status": 200, "encryptedFeatures": "abcdef123456GHIJKL0987654321..." } ``` You will need to decrypt the features first before passing into the SDK. Our front-end SDKs (Javascript and React) handle this for you automatically and we're in the process of adding built-in support to our other SDKs. --- ## Setting up a Bandit import MaxWidthImage from "@site/src/components/MaxWidthImage"; Bandits are run similarly to Experiments in GrowthBook. Key differences include: 1. We require you to select a **Decision Metric** to optimize towards 2. You have to decide the exploration window and update cadence for your Bandits ## Creating a Bandit ### 1. Selecting an Assignment Attribute ![Configuring Bandit Hash Attributes and Number of Variations](/images/bandits/bandit-config-1.png) First, select the assignment Attribute used to randomize users into variations and that maps to the metrics you will use to track user behavior. ### 2. Setting the Number of Variations Select the number of variations you want to set up. The Bandit works well in cases where you have 5 or more arms and you're looking to weed out poor performers quickly, but if you have too many arms for your amount of traffic, then it may take a long time for the Bandit to identify a winner. ### 3. Enable or Disable Sticky Bucketing [Sticky Bucketing](/app/sticky-bucketing) ensures that users are consistently assigned to the same variation even if they return to the experiment later and the variation weights have changed. Variation weights change frequently in a Bandit, so Sticky Bucketing is important if you want to ensure consistent user experiences. If you have Sticky Bucketing set up, we generally advise using it in your Bandit. Sticky bucketing helps keep user experiences fixed over time. For example, if your bandit arms represent different promotional values, it may be a bad user experience if a user returns to your website and sees a different promotional value. Sticky bucketing also permits you to learn from longer term outcomes, though bandits are generally not designed for this purpose. Finally, if sticky bucketing is disabled, the bandit uses only a user's first exposure and the metrics from that first exposure window. While this ensures that future exposures do not bias results, discarding data is not ideal. However, there are use cases where sticky bucketing is suboptimal. Sticky bucketing keeps users locked into their first exposure, which can hurt the reward of the bandit from switching users into better performing variations. Disabling sticky bucketing can reduce the cost of experimentation. If you do use Sticky Bucketing, you should set up your Sticky Bucketing service to store variations in the same place that you store and track your user identities. For example, if you plan on running a bandit using cookie-based identifiers, we strongly recommend using a cookie-based Sticky Bucketing approach to ensure that both the hash attribute and the sticky bucketing identifiers are kept in sync. Follow the instructions in the [Sticky Bucketing](/app/sticky-bucketing) guide to set up Sticky Bucketing in your SDK and enable it in GrowthBook. ![Configuring Bandit Decision Metric and Exploration Window](/images/bandits/bandit-config-2.png) ### 4. Set the exploration window and update cadence The Bandit needs to know how long it should collect data for before beginning to update variation weights. This is called the **Exploration Window**. After the Exploration Window, the bandit will begin to update variation weights at a set **Update Cadence**. ### 5. Selecting a Decision Metric The Decision Metric is the metric that the Bandit will optimize towards. After each **reweighting** update, the Bandit will assign more traffic to the variations that are performing better with respect to the Decision Metric. It should be a metric that you care about improving and that you can measure quickly after the user is exposed to the experiment. An ideal decision metric has the following properties: - It has a **short conversion window** and can be observed in a relatively short time frame. Short conversion windows allow the bandit to update with accurate information; a metric that takes longer to be observed will slow down Bandit learning. If sticky bucketing is disabled, it is important to use a short conversion window to avoid - If it is a proportion metric, the conversion rate should not be super low or super high. These extreme conversion rates can result in Bandits that are very slow to converge as a lot of data is needed to adequately separate the Bandit variations. ### 6. Specify the conversion window length Ideally your users convert in a window much shorter than your bandit's update cadence. This will help the bandit learn quickly and allocate future traffic to the best performing variations. If sticky bucketing is not enabled and your metrics are at the user level, this is extra important. If a user has multiple sessions, long conversion windows can result in double counting metric outcomes. If the user has multiple sessions across different bandit periods, then long conversion windows can attribute metric outcomes to the wrong variation. #### Exploration Window Your Exploration Window should be long enough that you get enough traffic into each arm before the Bandit begins optimizing traffic. We _require_ at least 100 users per variation before the bandit updates variation weights. You should make your Exploration Window long enough to: - Get a good number of conversions or metric values to begin estimating your Decision Metric. We recommend having at least 40 conversions per variation before you begin updating your bandit. If your conversion rate tends to be around 10%, then we recommend picking a window long enough to get 400 units per variation. - Get a decent mix of different units into your experiment (e.g. running a 8-hour exploration window may only capture users in particular time zones before it begins updating). #### Update Cadence The minimum update cadence you can set in GrowthBook Bandits is 15 minutes. However, because each update requires tracking events to flow from your users to your data warehouse, and then into GrowthBook via queries, in practice your data may take longer to materialize and the practical minimum update may be longer. Furthermore, the fewer units that enter your Bandit in each period, the less valuable it is to update the Bandit frequently. Unless you have a compelling reason to update your Bandit more frequently, we recommend setting your update cadence to daily or even longer, depending on the goals of your bandit. While longer update windows mean that the Bandit will send more traffic to lower performing variations, they also reduce the likelihood that a fluky day of traffic will cause undesirable weight updates. ### 7. Configure your different Variations After you create your bandit, you'll be able to configure the different variations each arm of the Bandit will serve. Use either Feature Flags, the Visual Editor, or URL Redirects to set up your different variations. You can also use the API to set up your variations. ![Linking Bandit Changes](/images/bandits/bandit-config-3.png) ### 8. Prerequisite: Double check SDK cache settings Bandits optimize towards winning variations by making frequent changes to variation weights. Therefore it is important to ensure that your SDK implementation is able to receive updates in a timely fashion. Most of our SDKs automatically cache bandit definitions locally to reduce network latency. While each SDK's default cache settings, namely the `maxAge` or `TTL` fields, are generally compatible with most bandit configurations, you should double-check your own configuration. If variation weights do not update in the SDK significantly more quickly than your bandit's [update cadence](#update-cadence), then you may experience slower bandit performance and even [Sample Ratio Mismatch (SRM)](/using/experimenting#sample-ratio-mismatch-srm). :::tip Cache max-age recommendations As a rule of thumb, try to aim for a cache `maxAge` or `TTL` that is at least _5 times faster_ than your **update cadence**. For a bandit with a daily update cadence, use an SDK cache max-age of 4-5 hours. For a bandit with an hourly update cadence, use a max-age of around 10 minutes. ::: - If you are using the [HTML Script Tag](/lib/script-tag), variation weights have a maximum staleness of 4 hours, which is sufficient for most scenarios. - Similarly, most of our SDKs have an internal cache with a default `maxAge` or `TTL` of 4 hours. - If you are applying any sort of network-level payload caching (such as a custom CDN or caching proxy), please ensure appropriate cache TTL and/or ejection policies are in place. --- ## Bandits import Pill from '@site/src/components/Pill'; import CommercialFeature from '@site/src/components/CommercialFeature'; ### What are Bandits? Bandits, also known as multi-armed bandits, are a type of experiment where the traffic assigned to each variation changes during the experiment. This is in contrast to standard experiments, where the percentages of traffic assigned to variations are static during the experiment. When your bandit is running, we recalculate the percent of traffic to allocate to each variation based on which variation (or arm) is performing better on a single **Decision Metric**. By driving more traffic to the best-performing variations, we can both: - Identify the best variation faster than standard experiments - Expose fewer users to poor-performing variations, which can help reduce negative impacts on metrics Bandits therefore are extremely powerful, but they come with trade-offs: - They require a single **Decision Metric** to optimize towards - They can be more complex to set up - They can introduce certain biases in estimating the true effect of variations - They can, in some cases, actually do worse at picking the best variation than traditional experimentation. [Get started with our Bandit setup guide!](/bandits/config) ### When should I run a Bandit? You should run a bandit when: - You want to reduce the cost of experimentation - You have a clear decision metric you are optimizing for - You have many arms you want to test against one another, and care less about learning about user behavior on several metrics ### Why should I run a Bandit? 1. Bandits can **reduce the cost of experimentation**. By allocating larger percentages of traffic to better performing arms, you avoid driving traffic to losing variations. 2. Bandits can quickly determine which variations are underperforming and direct traffic to the top performers, allowing you to **identify the best variation more quickly** than a standard experiment. With ten variations, a bandit may be able to quickly determine that several are poor performers. Sending traffic to the higher higher performing variations improves your ability to distinguish those high performers. ### Why should I run a standard Experiment instead? 1. Bandits require a **single decision metric** to optimize towards. If you have multiple metrics you care about, or if you want to understand the effect of each variation on multiple metrics, a standard Experiment may be better. Furthermore, a Bandit is more powerful if the decision metric is easy to measure right after experiment exposure. If you have a long sales cycle, or if you care about long-term effects, a standard Experiment may be better. 2. Bandits are known to **suffer from a [few types of _bias_](https://arxiv.org/abs/1905.11397)**. Because you adjust variation weights, stop early, and focus on the winners, Bandits can suffer from a variety of positive and negative biases when estimating variation averages. Because of this, standard Experiments can perform better in producing less biased estimates of experiment effects. We work hard to address some of these sources of bias in Bandits (e.g. by weighting across periods to deal with the fact that users entering your experiment on different days have different behavior), but it is still a limitation of the Bandit approach. 3. Bandits can even perform worse at selecting the best variation more quickly than traditional experiments in some cases, such as when there are only two arms. In that case, a standard Experiment can evenly split traffic between the two arms, which is best for reducing variance and increasing precision of lift estimates. 4. If you are not using [Sticky Bucketing](/app/sticky-bucketing), Bandits will cause users to change variations over time, which can be jarring for users if the feature being tested is not ephemeral, or easy to switch on and off for a given user. ### Comparison of Bandits and Experiments While Experiments and Bandits can be used in a host of overlapping cases, there are some rough guidelines for when to use each: | Characteristic | **Standard Experiments** | **Bandits** | | -------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Goal | Obtaining accurate effects and learning about customer behavior | Reducing cost of experimentation when learning is less important than just shipping the best version | | Number of Variations | Best for 2-4 | Best for 5+ | | Multiple goal metrics | Yes | No | | Changing Variation Weights | No | Yes | | Consistent User Assignment | Yes | Yes (with [Sticky Bucketing](/app/sticky-bucketing)) | ### GrowthBook's Bandit implementation GrowthBook's Bandit uses Thompson sampling, a Bayesian algorithm that balances _exploration_ (trying out different variations) and _exploitation_ (focusing on the best-performing variations). Thompson sampling allocates traffic proportionally to the probability that an arm is best. By sending some traffic to arms that are less likely to have the highest revenue, Thompson sampling _explores_ all variations when searching for a winner. Thompson sampling _exploits_ information about which arms are likely to be best. Sending more traffic to larger revenue arms saves money. Exploration vs exploitation is a fundamental tradeoff in adaptive experimentation. Proportional allocation in Thompson sampling nicely balances this tradeoff. In practice, GrowthBook ensures that each variation has at least 1% of traffic, in case your user behavior changes over time. ## FAQ 1. **Can I run a Bandit using the frequentist engine?** No. Currently Bandits are available only under the Bayesian engine, where Thompson sampling is easy to employ. 2. **How can I translate my frequentist p-value threshold into a success criterion for the Bayesian engine?** While the frequentist p-value and the Bayesian chance-to-win are very different concepts, they are often conceptualized similarly by their practitioners. One simple approach is to adjust your settings so that your Bayesian chance to win (default is 95%) is equal to 1 minus half of your p-value threshold. 3. **How can I use power analysis for a Bandit?** GrowthBook power results should be conservative for your Bandit. In many settings, Bandit power will be higher. However power in Bandits is much more complicated to estimate. 4. **What happens to users in worse-performing variants?** As the experiment progresses, worse-performing variations will receive progressively less traffic. New users will not be exposed to those worse-performing variations. The experience of returning users depends on whether you have set up and are using sticky bucketing; with sticky bucketing, users will continue to see the same variation they were originally assigned to, without sticky bucketing, users could be re-assigned when they return to the experiment. 5. **What’s the difference between multi-armed bandit and contextual bandit experiments?** GrowthBook uses multi-armed bandit experiments, which focus on optimizing traffic allocation among variations based purely on overall performance ([the Decision Metric](/bandits/config#3-selecting-a-decision-metric)). These experiments treat all users the same, without taking individual user characteristics into account. In contrast, contextual bandits use user-specific attributes (like location, device type, or past behavior) to select variations, aiming to tailor the experience to each user. While multi-arm bandit experiments aim for a global "best" variation, contextual bandits work to find the best variation for each unique user context. If you'd like to see contextual bandits in GrowthBook, [please let us know](https://slack.growthbook.io/?ref=bandits-doc)! --- ## Understanding Bandit Results import MaxWidthImage from "@site/src/components/MaxWidthImage"; ## Leaderboard The leaderboard is the best tool for comparing variations with respect to the decision metric. In the example below, the decision metric is revenue. ![Bandit Leaderboard](/images/bandits/bandit_leaderboard_1.png) Each row of the leaderboard contains information about one of the five variations. - The **Users** column on the leaderboard indicates the number of users per variation in the experiment. Note that the number of users is vastly unequal across variations, indicating that the bandit has been favoring some variations over others. - The **Mean** displays the variation mean for the decision metric. - The chart to the right of the columns displays credible intervals for mean revenue for each metric. - The black vertical lines indicate the estimated mean revenue for each variation and are equal to the values in the **Mean** column. - The colored bands around the black lines represent 95% credible intervals for mean revenue by variation. The leaderboard helps visualize how user traffic allocations connect to the variation means. In this example, Variation 2 has the smallest number of users (2,575) and the lowest mean revenue (\$31.95). The other variations are roughly similar in terms of mean revenue, with control being slightly ahead. When a variation hits a 95% probability of being the best variation (in other words, when the bandit would want to allocate 95% or more of traffic to a variation), we mark it as a winner in the leaderboard as the image below demonstrates. ![The leaderboard with a winner](/images/bandits/leaderboard-winner.png) ## Graphs of Performance Over Time Below the leaderboard are 3 different time series graphs to help you understand how the bandit evolved over time. ### Cumulative Variation Means The first graph shows the cumulative means of the decision metric over time. ![Cumulative means](/images/bandits/cumulative_means.png) On all graphs shown, the vertical lines correspond to times when the variation weights were updated. The overlap in the credible intervals in the leaderboard above causes mingling of the lines here. ### Probability of Winning The second graph shows how the probability of each variation being the best changes over time. ![Probability of winning](/images/bandits/prob_of_winning.png) The volume of color on the graph corresponds to the probability that each variation is the best. At the beginning of the experiment during the exploration phase, each variation had an equal chance of being the best. After the exploration phase, Variation 3 (in red) had the highest probability of being the best, and received the most traffic. Control and Variation 2 also had high probabilities of being the best, but received less traffic. Variations 1 and 4 had low probabilities of being the best and received the least traffic. ### Variation Weights The third graph shows the variation weights over time. ![Variation weights](/images/bandits/variation_weights.png) As in the time series plot above displaying probability of being the best, variations with larger variation weights have a larger proportion of color on the graph. The variation weight for a period is equal to the probability of being the best arm, unless an arm has less than a 1% chance of being the best, in which case it is set to 1% and all of the other weights are recalibrated. By mousing over the graph, the tool tip will display the variation weight and number of users for each variation at that time. This tooltip also works for the probability of winning graph. ## Explore Tab While bandits suffer from biased results, it may sometimes be of interest to look at differences in variations, as in a standard experiment. The explore tab shows standard experiment results for the decision metric and for any secondary or guardrail metrics. Results on the explore tab are interpreted in the same way as in a standard experiment. Again, we caution that these results may be biased, and should be interpreted with caution. ## Debugging Issues [Sticky bucketing](/app/sticky-bucketing) is a feature that ensures that users are consistently assigned to the same variation throughout the experiment. This is important for bandits, as it ensures that the same user is not assigned to different variations over time as the variation weights change. Sticky bucketing helps avoid multiple exposures, which can cause bias in the results as well as inconsistent user experiences. Evidence of multiple exposures likely indicates that your Sticky Bucketing is not set up at the same level as your randomization attribute for the Bandit. For more on this topic, see the section on [correctly set up sticky bucketing](/bandits/config#prerequisite-sticky-bucketing). ### What can cause multiple exposures? Imagine you are randomizing by a logged-in User ID. However, your sticky bucketing service is stored in a cookie. When the user first hits the bandit, we hash their User ID to a number from 0 to 1, and assign them the corresponding variation. We also store this in a cookie on the user's device. A day passes and the bandit updates the variation weights. Now, when the User ID returns on their phone, they get the same hash value from 0 to 1, but because the variation weights have changed, that value maps to a different variation. Furthermore, because they are on their phone, they do not have access to the cookie where their sticky bucket is stored. Then, they will be assigned to a new variation on their phone, triggering a multiple exposures warning. ## FAQ 1. **Why does the leaderboard show a negative value for the CI for a proportion metric?** GrowthBook's bandit uses solely a Gaussian Thompson sampling approach for both proportion and binomial metrics. This allows us to mitigate bias caused by conversion rates that change over time with a unified weighting approach. So you may see some negative values for proportion metrics, but they should normalize with increased sample size and if your conversion rates are less extreme (e.g. further from 0% and 100%). 2. **How can I see the number of conversions by arm?** You can see the number of _weighted_ conversions per arm on the Explore Tab. Because we apply a weighting approach to safe-guard the bandit against bias from changing conversion weights over time, the numerator used to compute means (either conversion rates for proportion metrics or just averages for mean metrics) is weighted over bandit update periods. --- ## Command Palette import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Command Palette The command palette is a keyboard-first launcher for everything in GrowthBook. Open it from any page to jump straight to a feature, experiment, metric, dashboard, saved group, navigation page, doc, or REST API endpoint. ![GrowthBook command palette open with a query and grouped results across pages, features, experiments, and metrics](/images/command-palette/command-palette-light.png) ## Open the command palette You can open it two ways: - Press `Cmd+K` on Mac or `Ctrl+K` on Windows and Linux. The shortcut works on every page and toggles the palette open or closed. - Click **Search** in the left sidebar. Press `Esc` or click outside the palette to close it. ## What you can search Start typing in the palette. Results group into the sections below, in this order: | Section | What it returns | Where it sends you | | ----------------- | ----------------------------------------------------------------------- | ------------------------------------ | | **Pages** | Sidebar destinations you have access to | The page in GrowthBook | | **Features** | Feature flags by ID and tags | The feature detail page | | **Experiments** | Experiments by name, description, and tags | The experiment detail page | | **Metrics** | Metrics, fact metrics, and metric groups by name, description, and tags | The metric or group page | | **Saved Groups** | Saved groups by name | The saved group page | | **Dashboards** | Product analytics dashboards by title | The dashboard page | | **Documentation** | Sections of the GrowthBook docs | docs.growthbook.io, in a new tab | | **REST API** | Endpoint groups in the public API reference | docs.growthbook.io/api, in a new tab | Each section shows up to five results. If a section has more matches, a **Show N more…** row reveals the rest. :::note Archived features, experiments, and metrics aren't indexed. Deleted dashboards are also excluded. ::: ## How matching works Pages, docs, and REST API rows use strict matching, so short queries route precisely. For example, typing `exp` surfaces the **Experiments** page first. Features, experiments, metrics, saved groups, and dashboards use fuzzy matching with smart tokenization. Identifiers are split on camelCase, kebab-case, and snake_case boundaries before matching, so `myFeatureFlag`, `my-feature-flag`, and `my_feature_flag` all match the query `feat flag`. Queries with minor typos still match. Results are scoped to your permissions and the current project. You only see pages and resources you can access. ## Keyboard shortcuts inside the palette - `↑` and `↓`: move between results. - `Enter`: open the selected result. Internal results open in the same tab. Documentation and REST API results open in a new tab. - `Esc`: close the palette. - Hovering with the mouse also moves the selection. :::tip Use the palette as a fast way into the API reference. Type a resource name like `fact metrics` or `sdk connections` and press `Enter` to jump straight to that section of the [REST API docs](https://docs.growthbook.io/api). ::: --- ## Compliance # Security Compliance GrowthBook takes security seriously and is compliant with **SOC2**, **GDPR**, and **HIPAA**. We perform regular penetration testing, have an active bug bounty program, and maintain strict controls to protect data privacy. View our [Trust Center](https://trust.growthbook.io/) to request access to the above certifications, as well as our most recent penetration test report, incident response plan, access control policy, and more. --- ## Setting up Contextual Bandits import CommercialFeature from '@site/src/components/CommercialFeature'; import Pill from '@site/src/components/Pill'; # Setting up a Contextual Bandit beta Contextual bandits are in closed beta for enterprise customers and will evolve with customer feedback. If you're interested in trying them out, please [contact us](https://growthbook.com/contact). This page is a checklist of everything you need in place before running your first [Contextual Bandit](/contextual-bandits/overview): upgrading to a compatible SDK, updating your tracking callback, and configuring GrowthBook prerequisites. ## Requirements at a glance - **An Enterprise plan** — Contextual Bandits are an Enterprise-only beta feature. - **A supported SDK on a compatible version** — currently JavaScript, React, and Node.js (see below). - **A Fact Metric to use as the Decision Metric** — legacy (non-fact) metrics are not supported as decision metrics. ## 1. Install or upgrade to a compatible SDK Contextual Bandit support ships in the following SDKs: | SDK | Package | Compatible versions | | ---------- | ------------------------------ | ------------------- | | JavaScript | `@growthbook/growthbook` | 1.7.0 or higher | | React | `@growthbook/growthbook-react` | 1.7.0 or higher | | Node.js | `@growthbook/growthbook` | 1.7.0 or higher | Install or upgrade to a compatible version with your package manager: ```bash npm install --save @growthbook/growthbook # React apps npm install --save @growthbook/growthbook-react ``` Note: for JS-based SDKs, 1.7 is a backwards-compatible release. :::note What happens on older SDKs? Contextual bandit rules degrade gracefully. SDKs below 1.7.0 (and all SDK languages without the capability) skip contextual bandit rules entirely and serve the feature's default value — they never bucket users with stale or global weights. This makes it safe to roll out a contextual bandit while part of your fleet is still on older SDK versions, but users on those versions won't enter the bandit. ::: ## 2. Check your SDK Connection Two things to verify on your SDK Connection, under **SDK Configuration → SDK Connections** in the GrowthBook app: 1. **Language and version** — make sure the connection's SDK language is JavaScript, React, or Node.js and its version is set to 1.7.0 or higher. GrowthBook only includes contextual bandit definitions in the SDK payload for connections that support them. 2. **Cache TTL** — contextual bandits change variation weights while running. If your SDK caches the payload longer than the bandit's update cadence, users will be bucketed with stale weights, which slows learning and can trigger SRM warnings. Make sure your SDK's `maxAge` / TTL settings refresh the payload significantly more often than the bandit reweights (or use streaming updates). ## 3. Update your tracking callback Contextual bandits personalize variation weights based on unit attributes that are set on the GrowthBook SDK. The best way to ensure we log the attributes used to bucket units is to use the new `trackingCallback` signature (with a new `user` argument) that allows you to pull the unit attributes that were used at evaluation time. Furthermore, for offline debugging and for future health checks in GrowthBook, consider tracking three additional fields: `leafId`, `banditVersion`, and `variationWeights`. These three fields are only set for contextual bandit assignments. ```js const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", trackingCallback: (experiment, result, user) => { analytics.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, // Directly pass through attributes used for contextual bandits (e.g. userRole) deviceId: user.attributes.deviceId, userRole: user.attributes.userRole, // Contextual bandit fields (undefined for regular experiments) leafId: result.leafId, banditVersion: result.banditVersion, variationWeights: result.variationWeights, }); }, }); ``` This change is optional only if you already track user attributes in your tracking callback. We strongly recommend passing the explicit `user.attributes` object and its fields to ensure the attributes being logged are the same as the ones being used to bucket users. ## 4. Create a Contextual Bandit Assignment Query To take advantage of the above attributes and additional contextual bandit fields, you need to create a dedicated **Contextual Bandit Assignment Query**. This query holds the assignment SQL plus the list of context columns the bandit is allowed to split on. Manage these on your GrowthBook Datasource page under **Contextual Bandit Assignment Queries**, or via the [REST API](https://docs.growthbook.io/api#tag/ContextualBanditAssignmentQueries/operation/createContextualBanditAssignmentQuery). The query must select one row per exposure event with the following columns: ```sql SELECT device_id, timestamp, experiment_id, variation_id, -- Context columns that match the new trackingCallback attribute -- columns you're using for personalization, e.g. userRole userRole, -- Optional contextual bandit health-check columns, logged by your tracking callback leaf_id, bandit_version, variation_weights FROM contextual_bandit_events ``` Requirements: - At least one context column (e.g. `userRole`) is required and ideally matches the columns you're sending from the `user` argument from your tracking callback. The set passed here must match your SDK Attributes in GrowthBook in order to be used to target the contextual bandit. - `leaf_id`, `bandit_version`, and `variation_weights` are the warehouse-side counterparts of the tracking callback fields from step 2. They are optional, but you should include them — later versions of contextual bandits will use them for health checks like SRM. ## 5. Create the Contextual Bandit Create the bandit under **Experiments → Contextual Bandits** in the left navigation. You'll need: - **A hash attribute** — the attribute used to randomize users into variations, which should map to the user identifier in your assignment query. For short-lived bandits, we suggest a `session_id` or `device_id` that will re-generate on future visits to ensure you get the most power while managing when a session ends yourself. - **A Contextual Bandit Assignment Query** — the query you created in step 4. The attribute columns you pass there will dictate the attributes that will be used to target the contextual bandit. Future versions of contextual bandits will let you select a subset of these columns to use as targeting attributes. - **Exploration window and update cadence** — how long the bandit collects data before it starts reweighting, and how often it reweights after that. See the [bandit configuration guide](/bandits/config#4-set-the-exploration-window-and-update-cadence) for guidance. - **A Decision Metric** — the single Fact Metric the bandit optimizes toward. The same guidance as for [multi-armed bandits](/bandits/config#5-selecting-a-decision-metric) applies: prefer a metric with a short conversion window and a conversion rate that isn't extremely low or high. This step can also be done via the REST API using the [createContextualBandit](https://docs.growthbook.io/api#tag/ContextualBandits/operation/createContextualBandit) endpoint. ## 6. Link a Feature Flag and start Once you've created a contextual bandit, you can link it to a feature flag and start it. A contextual bandit serves its variations through a **contextual bandit rule** on a feature flag, and needs at least one linked flag before it can start: 1. Create (or pick) a feature flag and link it from the bandit's page. This adds a contextual bandit rule to the flag in a draft revision. 2. Define the value each bandit variation should serve. 3. Start the bandit. Pending drafts on linked flags are published automatically when the bandit starts. Once running, GrowthBook refreshes the bandit on its update cadence: each refresh queries your warehouse, refits the context tree, and pushes updated per-leaf weights to your SDKs. You can watch the current weights and per-context results on the bandit's page. This step can also be done via the REST API by: - Creating a feature flag via the [postFeatureV2](https://docs.growthbook.io/api#tag/features-v2/operation/postFeatureV2) endpoint. - Linking the feature flag to the contextual bandit via the [addContextualBanditLinkedFeature](https://docs.growthbook.io/api#tag/ContextualBandits/operation/addContextualBanditLinkedFeature) endpoint. - Starting the contextual bandit via the [startContextualBandit](https://docs.growthbook.io/api#tag/ContextualBandits/operation/startContextualBandit) endpoint. ## Next steps - [Contextual Bandits overview](/contextual-bandits/overview) — concepts, and when to use one over a multi-armed bandit --- ## Driving a Contextual Bandit via API import CommercialFeature from '@site/src/components/CommercialFeature'; import Pill from '@site/src/components/Pill'; # Driving a Contextual Bandit via API beta This runbook walks through the full lifecycle of a Contextual Bandit driven entirely from the REST API. Every step assumes you have a valid Personal Access Token or Secret Key with the relevant Contextual Bandit permissions on the target project, and that the general prerequisites — a 1.7+ SDK, an updated tracking callback, and a connected data source — are already in place. See [Setting up a Contextual Bandit](/contextual-bandits/config) for that checklist. Contextual Bandits use two REST resources: - `/v1/contextual-bandit-queries` — the assignment SQL plus the targeting (context) columns the bandit splits on. This is the bandit-specific replacement for borrowing an Experiment Assignment Query off the datasource. - `/v1/contextual-bandits` — the bandit itself, which references a query by id. See the [REST API reference](https://docs.growthbook.io/api) for the full schema. :::info Contextual Bandits are an Enterprise-only feature. Every endpoint below returns `402 Plan Does Not Allow` on Pro / Free plans. ::: ## 1. Create a Contextual Bandit Query A Contextual Bandit needs a query that defines the assignment SQL and the targeting-attribute columns it splits on. `targetingAttributeColumns` is required and must be non-empty — a bandit with no context to split on is not a contextual bandit. Each named column must also appear in the query's `SELECT`. The query should also select the `leaf_id`, `bandit_version`, and `variation_weights` columns logged by your [tracking callback](/contextual-bandits/config#2-update-your-tracking-callback) — without them the SRM health check is skipped. ```bash curl -X POST 'https://api.growthbook.io/api/v1/contextual-bandit-queries' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "datasourceId": "ds_abc", "name": "Logged-in users", "userIdType": "user_id", "query": "SELECT user_id, timestamp, experiment_id, variation_id, leaf_id, bandit_version, variation_weights, country, device FROM experiment_viewed", "targetingAttributeColumns": ["country", "device"] }' ``` The response includes the new query id (`cbq_…`). You can manage queries with the standard CRUD endpoints (`GET /v1/contextual-bandit-queries`, `GET/PUT/DELETE /v1/contextual-bandit-queries/:id`); pass `?datasourceId=ds_abc` on the list endpoint to scope by datasource. ## 2. Create the Contextual Bandit Reference the query from the previous step via `contextualBanditQueryId`. ```bash curl -X POST 'https://api.growthbook.io/api/v1/contextual-bandits' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "name": "Homepage CTA bandit", "trackingKey": "homepage-cta-bandit", "datasource": "ds_abc", "contextualBanditQueryId": "cbq_abc", "decisionMetric": "metric_signup", "variations": [ { "key": "control", "name": "Control" }, { "key": "treatment", "name": "Treatment" } ], "contextualAttributes": ["country", "device"], "project": "proj_a" }' ``` Required fields are `name`, `trackingKey`, `datasource`, `contextualBanditQueryId`, `variations`, `decisionMetric`, and `contextualAttributes`. Other fields (`hashAttribute`, `queryFilter`, `activationMetric`, `minUsersPerLeaf`, `maxLeaves`, etc.) are optional and fall back to defaults. The response includes the new Contextual Bandit id (`cb_…`). The CB is provisioned in `status: "draft"` with an empty `currentLeafWeights` array at the document root; it stays empty until the first successful refresh. ## 3. Link a Feature Flag A Contextual Bandit serves its variations through a `contextual-bandit-ref` rule on a Feature Flag, and it needs at least one linked Feature Flag before it can start. Create the flag first with `POST /v2/features`, then link it: ```bash curl -X POST 'https://api.growthbook.io/api/v1/contextual-bandits/cb_xxx/linked-feature/homepage-cta' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "variations": [ { "variationId": "0", "value": "Sign up" }, { "variationId": "1", "value": "Get started free" } ] }' ``` `variations` must cover every Contextual Bandit variation exactly once — `variationId` is the `id` of each entry in the CB's `variations` array, not its `key`. Targeting (`condition`, saved groups, prerequisites, `coverage`) lives on the Contextual Bandit and is inherited by the rule, so those fields are not accepted here. The rule is appended to the bottom of the flag's rule list in a new draft revision, which auto-publishes when you start the Contextual Bandit. Pass `"autoPublish": true` to publish it right away, or `"draftVersion": 7` to add the rule to an existing draft instead of starting a new one. By default the rule applies to every environment; pass `"allEnvironments": false` with `"environments": ["production"]` to narrow it. Response: ```json { "featureId": "homepage-cta", "ruleId": "fr_abc123", "revisionVersion": 4, "published": false } ``` Two companion endpoints round this out: - `GET /v1/contextual-bandits/:id/linked-features` — the linked flags enriched with live/draft state, per-environment rule state, and variation values (the same payload the CB detail page renders). - `DELETE /v1/contextual-bandits/:id/linked-feature/:featureId` — removes every `contextual-bandit-ref` rule pointing at this bandit from the flag and drops the linkage. Like the POST, the removal lands in a draft unless you pass `?autoPublish=true`. When the flag has no such rule left, only the linkage is cleared. ## 4. Start the Contextual Bandit Move the CB out of `draft` (to `status: "running"`) so it is eligible for refresh: ```bash curl -X POST 'https://api.growthbook.io/api/v1/contextual-bandits/cb_xxx/start' \ -H 'Authorization: Bearer YOUR_API_KEY' ``` The response returns the updated CB: `{ "contextualBandit": { … } }`. ## 5. Refresh — run a snapshot Triggers the contextual-bandit snapshot pipeline. The orchestrator builds frozen settings, opens a ContextualBanditSnapshot (CBS) doc, runs the warehouse query and the stats engine, and on success persists a ContextualBanditEvent (CBE) with the new per-leaf weights. Each successful snapshot increments the CB's `banditVersion`. ```bash curl -X POST 'https://api.growthbook.io/api/v1/contextual-bandits/cb_xxx/refresh' \ -H 'Authorization: Bearer YOUR_API_KEY' ``` Response shape: ```json { "snapshotId": "cbs_abc", "cbeId": "cbe_def" } ``` `cbeId` is omitted when the run failed before producing an event (the CBS doc will be in `status: "error"`). ## 6. Read the latest results Returns the same payload the GrowthBook UI uses to render the CB results table — the latest stats engine output plus a snapshot-status summary. ```bash curl 'https://api.growthbook.io/api/v1/contextual-bandits/cb_xxx/results' \ -H 'Authorization: Bearer YOUR_API_KEY' ``` Response (abbreviated; `contextualBanditSnapshot` and `latest` are each `null` until a run exists): ```json { "contextualBanditSnapshot": { "attributes": ["country"], "responses": [ /* per-context stats rows (each tagged with its `leafId`) */ ], "leaf_map": [ /* one entry per decision-tree leaf; `context` is the AND of per-attribute clauses, e.g. { "leafId": 2, "context": [ { "attribute": "country", "levels": ["US", "UK"], "operator": "in" }, { "attribute": "browser", "levels": ["Chrome", "Firefox"], "operator": "not in" } ] } */ ] }, "latest": { "id": "cbs_abc", "status": "success", "error": "", "queries": [ /* QueryPointer[] */ ], "runStarted": "2026-05-29T12:00:00.000Z", "dateCreated": "2026-05-29T12:00:05.000Z", "multipleExposures": 0, "type": "standard", "triggeredBy": "manual" } } ``` `latest.status` is one of `running`, `success`, or `error`, and `latest.runStarted` may be `null` if the run hasn't started. Use the companion endpoints for finer-grained inspection: - `GET /v1/contextual-bandits/:id/snapshots` — list recent snapshot runs (optional `?limit=`, max 100). - `GET /v1/contextual-bandits/:id/snapshots/:snapshotId` — single snapshot status. - `GET /v1/contextual-bandits/:id/events` — list CBE outputs, one per successful snapshot (optional `?limit=`, max 100). - `GET /v1/contextual-bandits/:id/events/:eventId` — single CBE. - `GET /v1/contextual-bandits/:id/current` — current root-level `currentLeafWeights` plus the latest event object (`latestEvent`, or `null`). The `/current` response looks like: ```json { "currentLeafWeights": [ { "leafId": 0, "condition": { "country": "US" }, "weights": [ { "variationId": "0", "weight": 0.38 }, { "variationId": "1", "weight": 0.62 } ] }, { "leafId": 1, "condition": { "country": { "$in": ["US", "UK"] } }, "weights": [ { "variationId": "0", "weight": 0.41 }, { "variationId": "1", "weight": 0.59 } ] }, { "leafId": 2, "condition": { "browser": { "$nin": ["Chrome", "Firefox"] } }, "weights": [ { "variationId": "0", "weight": 0.55 }, { "variationId": "1", "weight": 0.45 } ] } ], "latestEvent": { "id": "cbe_def", "contextualBandit": "cb_xxx", "snapshotId": "cbs_abc", "weightsWereUpdated": true, "degreesOfFreedom": 3, "dateCreated": "2026-05-29T12:00:05.000Z" } } ``` ## 7. Stop the Contextual Bandit When you're done, stop the CB: ```bash curl -X POST 'https://api.growthbook.io/api/v1/contextual-bandits/cb_xxx/stop' \ -H 'Authorization: Bearer YOUR_API_KEY' ``` Like `start`, this returns the updated CB under `contextualBandit`. ## Standard CRUD Both resources also expose the default CRUD endpoints: - `GET /v1/contextual-bandits` (filter with `?projectId=`, `?datasourceId=`, or `?trackingKey=`), `GET /v1/contextual-bandits/:id`, `POST /v1/contextual-bandits`, `PUT /v1/contextual-bandits/:id`, `DELETE /v1/contextual-bandits/:id`. - `GET /v1/contextual-bandit-queries` (filter with `?datasourceId=`), `GET /v1/contextual-bandit-queries/:id`, `POST /v1/contextual-bandit-queries`, `PUT /v1/contextual-bandit-queries/:id`, `DELETE /v1/contextual-bandit-queries/:id`. ## Permissions cheat sheet Permissions are checked directly against the Contextual Bandit doc (CBs no longer delegate RBAC to a paired experiment): - **Read** (`GET results`, `GET current`, `GET snapshot(s)`, `GET event(s)`, `GET /:id`, `GET /`) — `canReadSingleProjectResource(cb.project)`. - **Create** (`POST /`) — `canCreateContextualBandit(cb)`. - **Update** (`PUT /:id`) — `canUpdateContextualBandit(existing, updated)`. - **Run** (`POST refresh`, `POST start`, `POST stop`) — `canRunContextualBandit(cb, environments)`. - **Delete** (`DELETE /:id`) — `canDeleteContextualBandit(cb)`. - **Link / unlink a Feature Flag** (`POST` / `DELETE /:id/linked-feature/:featureId`) — `canUpdateContextualBandit` on the bandit, plus `canUpdateFeature`, `canManageFeatureDrafts`, and `canPublishFeature` (scoped to the rule's environments) on the Feature Flag. Contextual Bandit Query create/update/delete are gated by the datasource: you need `canUpdateDataSourceSettings` on the query's datasource. Plan gate (`hasPremiumFeature("contextual-bandits")`) is checked first on every endpoint and short-circuits with `402` before any data access. --- ## Contextual Bandits import CommercialFeature from '@site/src/components/CommercialFeature'; import Pill from '@site/src/components/Pill'; # Contextual Bandits beta ### What are Contextual Bandits? A Contextual Bandit is a [bandit](/bandits/overview) that personalizes which variation a user sees based on their **context** — attributes like country, device, or past behavior. Like a standard multi-armed bandit, traffic weights change while the experiment runs to favor better-performing variations. The difference is that a contextual bandit learns a _separate_ set of weights for each context, instead of a single global set of weights for all users. In other words, a multi-armed bandit asks "which variation is best?" A contextual bandit asks "which variation is best **for this kind of user**?" Where a global winner would be applied to everyone, a contextual bandit can send users in one country to one variation and users in another country to a different one — whatever performs best on the **Decision Metric** within each context. ### When should I run a Contextual Bandit? You should run a contextual bandit when: - You expect the **best variation to differ across user segments** (e.g. by region, device, or plan), rather than there being one global winner. - You have a single, clear **Decision Metric** to optimize toward. - You can express the segmenting attributes as **columns in your assignment query** so the bandit can split on them. - You care more about shipping the best experience to each segment than about producing unbiased, per-metric learnings. ### How is it different from a multi-armed bandit? | Characteristic | **Multi-Armed Bandit** | **Contextual Bandit** | | --------------- | ------------------------------------------- | ------------------------------------------------------ | | Goal | Find the single best variation for everyone | Find the best variation for each user context | | Traffic weights | One global set of weights | A separate set of weights per context | | Inputs | Decision Metric only | Decision Metric **plus** targeting/context attributes | | Best when | One variation is best for all users | The best variation varies across segments | | Setup | Standard assignment query | Assignment query that also selects the context columns | ### What you need to run one A contextual bandit splits traffic on **context attributes**, so it needs an assignment query that selects those columns in addition to the user identifier and timestamp. In GrowthBook these are defined on a **Contextual Bandit Query**, which holds the assignment SQL plus the list of `targetingAttributeColumns` the bandit is allowed to split on. Each context column you want to use must appear in the query's `SELECT`, and at least one targeting attribute is required — a bandit with no context to split on is just a multi-armed bandit. To get started you will need: - A connected **Data Source** with a warehouse the bandit can query. - An assignment query that selects your user id, timestamp, and one or more **context columns**. - A single **Decision Metric** to optimize toward. - Two or more **variations** to test. - An **SDK that supports contextual bandits** — version 1.7.0+ of the JavaScript, React, or Node.js SDK. For a step-by-step checklist covering SDK installation, tracking callback changes, and the rest of the setup, see [Setting up a Contextual Bandit](/contextual-bandits/config). ### GrowthBook's Contextual Bandit implementation Like multi-armed bandits, contextual bandits use **Thompson sampling**, a Bayesian algorithm that balances _exploration_ (trying variations to learn how they perform) and _exploitation_ (sending more traffic to the variations that look best). The key addition is that GrowthBook fits a decision tree over your context attributes, partitioning users into **leaves** — groups that share similar context — and then runs Thompson sampling _within each leaf_. This is how the bandit can converge to different winners for different kinds of users. As with standard bandits, GrowthBook ensures every variation keeps at least a small share of traffic within each context, so the bandit can keep adapting if user behavior changes over time. The statistical details are covered in the [Contextual Bandit technical reference](/statistics/contextual-bandit-technical). ## FAQ 1. **When is a contextual bandit worth it over a multi-armed bandit?** Only when you have a real reason to believe the best variation differs across user segments. If one variation is best for everyone, a contextual bandit adds complexity (splitting your data across contexts, which needs more traffic per context) without a payoff. A plain multi-armed bandit is the better default. 2. **Can I run a Contextual Bandit using the frequentist engine?** No. Like multi-armed bandits, contextual bandits are available only under the Bayesian engine, where Thompson sampling is used. 3. **What happens if a context has very little traffic?** GrowthBook controls how finely it splits using settings like `minUsersPerLeaf` and `maxLeaves`. 4. **Do contextual bandits suffer from the same biases as multi-armed bandits?** Yes — because traffic weights change adaptively, the same [adaptive-experimentation biases](https://arxiv.org/abs/1905.11397) apply, and splitting by context can make them more pronounced in low-traffic segments. If unbiased per-metric effect estimates are your goal, a standard experiment is still the better tool. --- ## GrowthBook and Google Analytics GA4 import MaxWidthImage from "@site/src/components/MaxWidthImage"; # A/B Testing with Google Analytics 4 (GA4) and GrowthBook This guide walks you through using GrowthBook with Google Analytics 4 (GA4) for A/B testing. There are a few parts to this, connecting GA4 to BigQuery, connecting GrowthBook to BigQuery, and then configuring GrowthBook to track correctly with GA4. For help debugging GA4 integration issues, please review our [GA4 Troubleshooting Guide](/kb/google-analytics/google-analytics-ga4-troubleshooting). You can watch a video version of this guide, largely focused on Google Optimize replacement, here: ## Configuring GrowthBook to use Google Analytics GA4 as a data source GrowthBook connects to Google Analytics (GA4) via BigQuery. This process is straight forward, and outlined below. You can also find Google's documentation on how to create this link [here](https://support.google.com/analytics/answer/9823238). ### 1. Create a BigQuery Project (if you don't have one) If you don't have a BigQuery project, you'll need to create one. Go to your [Google Cloud Console](https://console.cloud.google.com/) and create a new project: ![GA4 - BigQuery create project](/images/guides/GA4-1-project-list.png) Click on **_Create new project_** from the right and give your project a name. ![GA4 - BigQuery create and name your project](/images/guides/GA4-2-create-project.png) Once created, you'll be redirected to the BigQuery dashboard. ![GA4 - BigQuery new project dashboard](/images/guides/GA4-3-new-project.png) If you created a new project, the BigQuery API is automatically enabled. Otherwise, you'll need to [enable it manually](https://console.cloud.google.com/flows/enableapi?apiid=bigquery) :::note Using the BigQuery sandbox project If you are just testing GrowthBook with GA4 out, you can use the sandbox project that Google provides for free. When you create a new cloud project the sandbox should be automatically enabled. You can find more information about the sandbox [here](https://cloud.google.com/bigquery/docs/sandbox). ::: ### 2. Connect Google Analytics to BigQuery Log into your Google Analytics account and navigate to the Admin section. From there, make sure you have selected the right property, and scroll down to **_Product Links_** section. Click on the menu named **_BigQuery Links_** ![GA4 BigQuery new project dashboard](/images/guides/GA4-4-link-to-bigquery-1.png) Click on the **_Link_** button. This will open a menu that allows you select the project. Select on the **_Choose a BigQuery Project_** link. ![GA4 BigQuery, choose a project](/images/guides/GA4-5-link-to-bigquery-2.png) Select the project you wish to send your GA4 data to: ![GA4 BigQuery connect to project](/images/guides/GA4-5-link-to-bigquery-3.png) Then click **Next**. ![GA4 BigQuery connected to project](/images/guides/GA4-5-link-to-bigquery-4.png) On the next step you'll be presented with some options about the connection. ![GA4 BigQuery link options](/images/guides/GA4-5-link-to-bigquery-5.png) Here you can choose the frequency of data updates, either daily or streaming. To use Streaming, you'll need a BigQuery account with billing info added. Depending on your use case, daily updates may be sufficient. On the final step you'll be asked to confirm your choices. When finished, you should see something like this, verifying that the connection was successful. ![GA4 BigQuery link created](/images/guides/GA4-5-link-to-bigquery-6.png) And then your BigQuery link will show up on the listing page: ![GA4 BigQuery successfully connected](/images/guides/GA4-5-link-to-bigquery-7.png) :::note Loading Google Analytics via Google Tag Manager (GTM) If you are loading Google Analytics via Google Tag Manager (GTM), you may need to add the custom event to GTM to ensure the data is passed to GA4 from the datalayer. You can add the custom event by following this [section from the GTM guide](/guide/google-tag-manager-and-growthbook#4-tracking-via-datalayer-and-gtm). Our [Google Tag Manager (GTM) Troubleshooting Guide](/kb/google-analytics/google-tag-manager-gtm-troubleshooting) may be helpful for debugging if needed. ::: ### 3. Configure BigQuery for GrowthBook You'll need to give GrowthBook permissions to your BigQuery project so that we can access the data. We have created a guide just for this, which you can find [here](/guide/bigquery) ### 4. Connect GrowthBook to BigQuery Within GrowthBook, navigate to the **_Analysis_** section, and then click on the **_Data Sources_** page. Add a new data source, and select **_Google Analytics (GA4)_**. ![GA4 GrowthBook connect to GA4](/images/guides/GA4-6-add-GA4-datasource.png) ![GA4 GrowthBook connect to GA4](/images/guides/GA4-6-add-GA4-datasource2.png) Then add your BigQuery connection info. GrowthBook will pre-populate the SQL queries required to use your GA4 data. You can also add a custom SQL query if you want to use a different table or filter the data in some way as you like. :::note Automatic SQL query creation While GrowthBook will pre-populate the SQL queries for you, you may need to adjust the experiment query to match your data depending on the way you are tracking your experiments (see the **_trackingCallback_** below). ::: Once connected, you can add any additional metrics or dimensions, and then you can use your GA4 data for your experiments. You can use all your existing events and tracking- GrowthBook only requires one additional tracking call when a user is exposed to an experiment. ## Running experiments with GrowthBook and GA4 With the data source connected, you can integrate the GrowthBook SDK into your application to run A/B tests. Once implemented, the SDK will do the random assignments and send the experiment exposure event to GA4 based on the settings in the GrowthBook UI. :::info Visual editor for creating experiments We do have a visual editor for creating experiments as part of our Pro plan. Our visual editor is meant for simple experiments. Experiments that are more complex are best created by writing code with feature flags. ::: ### SDK integration for GA4 The easiest and recommended way to integrate GrowthBook is by using our [Script Tag SDK](/lib/script-tag). This SDK will work out-of-the-box with GA4 without any configuration required. :::note Debugging GA4 integrations If your experiment is not firing the `trackingCallback` you can use our [Chrome developer tool](https://chrome.google.com/webstore/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) to help you debug and make sure the user attributes are being set correctly. ::: Implementing the experiment variations can be done with code with inline experiments, using the feature flags, or by using our visual editor. ## Troubleshooting Please see our our [GA4 Troubleshooting Guide](/kb/google-analytics/google-analytics-ga4-troubleshooting) for help debugging GA4 integrations with GrowthBook. If relevant, see our [GTM Setup Guide](/guide/google-tag-manager-and-growthbook) and our [Google Tag Manager (GTM) Troubleshooting Guide](/kb/google-analytics/google-tag-manager-gtm-troubleshooting). --- ## Use environment variables or similar for passwords! ## Configuration Settings Once you have chosen your event tracker and data source type and successfully connected, you will be given an opportunity to modify your configuration settings. For many applications GrowthBook will have chosen the correct configuration settings straight out of the box based upon which event tracker you choose. In some instances you may need to tweak them slightly, or in the case of using a custom datasource, define them more explicitly. ### Identifier Types These are all the types of identifiers you use to split traffic in an experiment and track metric conversions. Common examples are `user_id`, `anonymous_id`, `device_id`, and `ip_address`. ### Experiment Assignment Queries An experiment assignment query returns which users were part of which experiment, what variation they saw, and when they saw it. Each assignment query is tied to a single identifier type (defined above). You can also have multiple assignment queries if you store that data in different tables, for example one from your email system and one from your back-end. The end result of the query should return data like this: | user_id | timestamp | experiment_id | variation_id | | ------- | ------------------- | -------------- | ------------ | | 123 | 2021-08-23-10:53:04 | my-button-test | 0 | | 456 | 2021-08-23 10:53:06 | my-button-test | 1 | The above assumes the identifier type you are using is `user_id`. If you are using a different identifier, you would use a different column name. Here's an example query you might use: ```sql SELECT user_id, received_at as timestamp, experiment_id, variation_id FROM events WHERE event_type = 'viewed experiment' ``` Make sure to return the exact column names that GrowthBook is expecting. If your table’s columns use a different name, add an alias in the SELECT list (e.g. `SELECT original_column as new_column`). #### Duplicate Rows If a user sees an experiment multiple times, you should return multiple rows in your assignment query, one for each time the user was exposed to the experiment. This helps us detect when users were exposed to more than one variation, and eventually may be useful in helping build interesting time series. #### Experiment Dimensions In addition to the standard 4 columns above, you can also select additional dimension columns. For example, `browser` or `referrer`. These extra columns can be used to drill down into experiment results. #### Identifier Join Tables If you have multiple identifier types and want to be able to auto-merge them together during analysis, you also need to define identifier join tables. For example, if your experiment is assigned based on `device_id`, but the conversion metric only has a `user_id` column. These queries are very simple and just need to return columns for each of the identifier types being joined. For example: ```sql SELECT user_id, device_id FROM logins ``` #### SQL Template Variables Within your queries, there are several placeholder variables you can use. These will be replaced with strings before being run based on your experiment. This can be useful for giving hints to SQL optimization engines to improve query performance. The variables are: - **startDate** - `YYYY-MM-DD HH:mm:ss` of the earliest data that needs to be included - **startYear** - Just the `YYYY` of the startDate - **startMonth** - Just the `MM` of the startDate - **startDay** - Just the `DD` of the startDate - **startDateUnix** - Unix timestamp of the startDate (seconds since Jan 1, 1970) - **endDate** - `YYYY-MM-DD HH:mm:ss` of the latest data that needs to be included - **endYear** - Just the `YYYY` of the endDate - **endMonth** - Just the `MM` of the endDate - **endDay** - Just the `DD` of the endDate - **endDateUnix** - Unix timestamp of the endDate (seconds since Jan 1, 1970) - **experimentId** - Either a specific experiment id OR `%` if you should include all experiments For example: ```sql SELECT user_id, anonymous_id, received_at as timestamp, experiment_id, variation_id FROM experiment_viewed WHERE received_at BETWEEN '{{ startDate }}' AND '{{ endDate }}' AND experiment_id LIKE '{{ experimentId }}' ``` **Note:** The inserted values do not have surrounding quotes, so you must add those yourself (e.g. use `'{{ startDate }}'` instead of just `{{ startDate }}`) ### Jupyter Notebook Query Runner This setting is only required if you want to export experiment results as a Jupyter Notebook. There is no one standard way to store credentials or run SQL queries from Jupyter notebooks, so GrowthBook lets you define your own Python function. It needs to be called `runQuery`, accept a single string argument named `sql`, and return a pandas data frame. Here's an example for a Postgres (or Redshift) data source: ```python import os import psycopg2 import pandas as pd from sqlalchemy import create_engine, text # Use environment variables or similar for passwords! password = os.getenv('POSTGRES_PW') connStr = f'postgresql+psycopg2://user:{password}@localhost' dbConnection = create_engine(connStr).connect(); def runQuery(sql): return pd.read_sql(text(sql), dbConnection) ``` **Note:** This python source is stored as plain text in the database. Do not hard-code passwords or sensitive info. Use environment variables (shown above) or another credential store instead. ## Schema Browser When you connect a supported data source to GrowthBook, we automatically generate metadata that is used by our Schema Browser. The Schema Browser is a user-friendly interface that makes writing queries easier as you can easily explore information about the datasource such as databases, schemas, tables, columns, and data types. ![GrowthBook Schema Browser](/images/growthbook-schema-browser.png) Below are the data sources that currently support the Schema Browser: - AWS Athena - _Requires a Default Catalog_ - BigQuery - _Requires a Project Name and Default Dataset_ - ClickHouse - Databricks - _Currently only supported on version 10.2 and above with a Unity Catalog_ - MsSQL/SQL Server - MySQL/MariaDB - Postgres - PrestoDB (and Trino) - _Requires a Default Catalog_ - Redshift - Snowflake --- ## Setting up Amplitude as an event tracker import DefaultInstructions from './_default-event-instructions.mdx' export const { title, description, sidebar_label, slug } = frontMatter # Amplitude Event Tracker Setup ## Exposure Tracking For client side testing, you can fire a custom track event from the GrowthBook SDK's `trackingCallback` function, like so: ```js trackingCallback: (experiment, result) => { amplitude.track('Experiment Viewed', {experimentId: experiment.key, variantId: result.key}); } ``` You can also use Amplitudes own experiment javascript format: ```js trackingCallback: (experiment, result) => { amplitude.track('$exposure', {flag_key: experiment.key, variant: result.key}); } ``` You can read more about Amplitude's exposure event [here](https://www.docs.developers.amplitude.com/experiment/general/exposure-tracking/) ## Integrating with Amplitude Data For GrowthBook to integrate with Amplitude you first must export your Amplitude data to the data warehouse of your choice. You can read more about it on [Amplitudes help pages](https://www.docs.developers.amplitude.com/data/destinations/). Currently, Amplitude supports exporting to Redshift, Snowflake, BigQuery, and S3 (and use Athena). Once you have set up a data export to one of these destinations, you can connect GrowthBook to it. :::note No automatic SQL query creation GrowthBook won't automatically create SQL queries for Amplitude, but you can follow the general instructions below. If you need any help, [reach out to us](https://slack.growthbook.io) and we can help ::: --- ## Setting up CleverTap as an event tracker import MissingEventTrackerDocs from './missing-event-tracker-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## Setting up a custom data source GrowthBook just needs you to write a couple SQL queries in order to query your data. Writing this SQL is a (mostly) one-time setup task. After building out this library of queries, they can easily be reused across many experiments. Don’t worry about the potentially huge number of rows returned by these raw queries. They are never run directly as-is and are instead combined, filtered, and aggregated as part of larger queries. Most of the final queries run by GrowthBook result in fewer than 10 rows returned. In the spirit of transparency, any time a query is run, you should see a `View Queries` link in the app to view the raw SQL sent to the data warehouse. This can help with debugging or let you move a query into a tool like Mode Analytics for more advanced analysis. Find more info on what needs to be configured [here](/app/datasources#configuration-settings). --- ## Setting up Firebase as an event tracker import MissingEventTrackerDocs from './missing-event-tracker-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## Setting up Freshpaint as an event tracker import DefaultInstructions from './_default-event-instructions.mdx' export const { title, description, sidebar_label, slug } = frontMatter # Freshpaint Event Tracker ## Event Tracker Setup For JS, you can track the experiment exposure events by adding the following code the `trackingCallback` in the GrowthBook SDK. ```js trackingCallback: (experiment, result) => { freshpaint.track('Experiment Viewed', {experimentId: experiment.key, variantId: result.key}); } ``` ## Data Source Setup Freshpaint does not support direct data access, instead, to work with GrowthBook, you'll need to export your data to a data warehouse. You can read about the supported data destinations [here](https://documentation.freshpaint.io/destinations/warehouses). Freshpaint lets you export to BigQuery, Databricks, Postgres, Redshift, and Snowflake. :::note Automatic SQL query creation GrowthBook will attempt automatically create SQL queries for Freshpaint, but this may require adjustments. General instructions for setting up SQL sources are below. If you need any help, [reach out to us](https://slack.growthbook.io) and we can help ::: --- ## Setting up Fullstory as an event tracker import MissingEventTrackerDocs from './missing-event-tracker-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## Setting up Heap as an event tracker import MissingEventTrackerDocs from './missing-event-tracker-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## Setting up Jitsu as an event tracker import MissingEventTrackerDocs from './missing-event-tracker-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## Setting up KeenIO as an event tracker import MissingEventTrackerDocs from './missing-event-tracker-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## GrowthBook and Matomo import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Configuring GrowthBook to work with Matomo GrowthBook makes it easy to do A/B testing using Matomo as an event source. This guide shows you how to set up GrowthBook to work with self-hosted Matomo and the MySQL/MariaDB that it runs on. ## 1. Integrate GrowthBook's SDK with Matomo within your application You can follow the SDK integration guides depending on your language. We need to add the `TrackingCallback` call specifically for Matomo. For this guide, we'll use Javascript (Next.js). Matomo lets you track custom events, and you can adjust the `Event Category`, `Event Action`, `Event Name`, and `Event Value`. You can read more about the custom tracking call at [Matomo's documentation site](https://matomo.org/faq/reports/implement-event-tracking-with-matomo/). :::caution Caveats with Matomo's event schema Event Category cannot contain any spaces. {" "} If Event Name is 0 or "0" it will record null. Event Value must be an integer. ::: To work with the default data schema in GrowthBook, we will encode the following: - `Event Category`: will have the value `ExperimentViewed` to isolate the exposure events. - `Event Action`: will store the experiment key. - `Event Name`: will store the variation id (with a prefix of `v` or some other string) ```ts // Create a GrowthBook instance const growthbook = new GrowthBook({ trackingCallback: (experiment, result) => { window["_paq"] = window._paq || []; window._paq.push([ "trackEvent", "ExperimentViewed", experiment.key, "v" + result.variationId, ]); }, }); ``` :::info Event Name prefix for Matomo We use the string prefix on the Event Name so that it is saved correctly, as `0` saves as `null`, but `v0` works correctly. This prefix is stripped with the SQL exposure query in step two. ::: With that set, we need to load the user id into the GrowthBook user attributes. The following code shows how to do this with the Matomo anonymous `visitor ID` ```ts // add the Matomo anonId when loaded let visitor_id; if ("_paq" in window) { _paq.push([ function () { visitor_id = this.getVisitorId(); growthbook.setAttributes({ ...growthbook.getAttributes(), id: visitor_id, }); }, ]); } ``` For Next.js, we would need to do this in the useEffect and also detect the window in case of a server side render: ```tsx // Create a GrowthBook instance const growthbook = new GrowthBook({ trackingCallback: (experiment, result) => { if (window) { window["_paq"] = window._paq || []; window._paq.push(['trackEvent', 'ExperimentViewed', experiment.key, "v"+result.variationId]); } } }); export default function MyApp({ Component, pageProps }) { useEffect(() => { // Load feature definitions from API fetch(process.env.NEXT_PUBLIC_GROWTHBOOK_FEATURES_URL) .then((res) => res.json()) .then((json) => { growthbook.setFeatures(json.features); }); // TODO: replace with real targeting attributes growthbook.setAttributes({ "company": "foo", "browser": "foo", "url": "foo" }); // add the Matomo anonId when loaded let visitor_id; if("_paq" in window) { _paq.push([function () { visitor_id = this.getVisitorId(); growthbook.setAttributes({ ...growthbook.getAttributes(), id: visitor_id }); }]); } }, []); ... ``` ## 2. Add the Data Source for Matomo From within the GrowthBook application, navigate to the `Data Sources` page from within the `Analysis` section. Click on `add data source` and select `Matomo` from the list of event sources. ![Add Matomo data source](/images/new-data-sources-modal.png) After you select Matomo, you will be prompted to connect to your MySQL/Maria database. ![Add MySQL/MariaDB data source](/images/guides/matomo-2-connect-to-db.png) Once you've completed the connection GrowthBook will and then you can adjust some settings specific to your instance and how you implemented the Matomo tracking code. ![Setting Matomo options](/images/guides/matomo-3-matomo-options.png) When you save, you'll have the experiment exposure queries for anonymous and user_id set for you. ![Default Matomo experiment exposure query](/images/guides/matomo-4-matomo-anon.png) You can test running this query directly against your database. It should return a list of experiments exposure events. You will still need to define the metrics you want to test against from the metrics settings. --- ## Missing Event Tracker Docs import Link from '@docusaurus/Link'; export default function MissingEventTrackerDocs({ eventTracker, slug }) { return ( Growthbook will modify its default sql settings to work as seamlessly as possible with {eventTracker}. After choosing {eventTracker} as your data source and filling in the connection info, you can proceed with any modifications of your configuration settings. Unfortunately a detailed guide for {eventTracker} has not been written yet. However we do have a thriving developer community on Slack that you can reach out to in order to get your implementation working. Once you get your implementation working please consider contributing back to the community by creating this page. ); } --- ## Using GrowthBook with Mixpanel :::warning Deprecation warning Using Mixpanel as a direct datasource is no longer supported. This is because because Mixpanel has placed JQL, their query language, in [maintenance mode](https://docs.mixpanel.com/docs/reports/apps/jql). In order to use Mixpanel with GrowthBook, we suggest you set up an export to a data warehouse following [Mixpanel's documentation](https://docs.mixpanel.com/docs/data-pipelines) and then use that separate data warehouse (e.g. Snowflake, BigQuery) as your Data Source in GrowthBook. ::: --- ## Setting up MParticle as an event tracker import MissingEventTrackerDocs from './missing-event-tracker-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## GrowthBook and RudderStack import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Configuring GrowthBook to work with RudderStack GrowthBook supports RudderStack as an event source for easily adding A/B testing to your application. This document is a how-to guide for setting up RudderStack with your application, connecting RudderStack to BigQuery and then connecting GrowthBook to this data. ## 1. Setup RudderStack If you have already set up RudderStack, you can skip this step. RudderStack allows you to send events from your application to any data destination, including BigQuery, which is what we're going to use it for here. Once you have a RudderStack account, you need to create a BigQuery bucket for the event data to be stored. You can follow the [directions from RudderStack](https://www.rudderstack.com/docs/destinations/storage-platforms/google-cloud-storage/), but the short version is: - From the Google Cloud console for BigQuery, create a new bucket (or you can also use an already existing bucket). - Create a new service account under [IAM & Admin → Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) for RudderStack to connect to this bucket. - Give Storage `Object Admin` role access to the newly created service account. - Under the `actions` menu on the service account you just created, click on manage keys to create a new key. Choose key type as JSON. This will cause a JSON key file to be downloaded to your computer. With the bucket created, we can create a RudderStack warehouse destination. ![Add a data warehouse destination for RudderStack](/images/guides/rudderstack-1-addwarehouse.png) ![Select BigQuery from the list](/images/guides/rudderstack-2-adddestination.png) ![Add the BigQuery account information](/images/guides/rudderstack-3-addbucket.png) The `Project` and `bucket name` needs to match the values you set up when creating the Google cloud project and bucket. Once you've created the destination, you should see it successfully connect ![Successfully added the BigQuery account information](/images/guides/rudderstack-4-connection-completed.png) And it will show up as a connected destination on your RudderStack Connect dashboard. ![BigQuery connection in rudderstack](/images/guides/rudderstack-4.5-bigQueryDestination.png) ## 2. Connect GrowthBook Now that we have data flowing into BigQuery, we need to have GrowthBook connect to it. It is best practice to create another service account for GrowthBook to connect to BigQuery. (We have a complete guide on how to do this here: [Setting up BigQuery as a data source](/guide/bigquery)) Navigate to Analysis → Data Sources from within GrowthBook and click on 'Add Data Source' Button. From the window that pops up, select RudderStack: ![Choose RudderStack data schema](/images/new-data-sources-modal.png) After choosing RudderStack you'll be presented with a connection window to connect to BigQuery. ![Connect to your database](/images/guides/rudderstack-add-bigquery.png) When successfully connected, you'll be redirected to the data source page. You'll see all the connection options for BigQuery as well as the queries used. By choosing RudderStack as you added the data source, GrowthBook will pre-populate the experiment exposure query which is need to determine which user saw which experiment variation. Depending on your needs, you may still need to adjust these queries - for instance if you're tracking both `anonymous_id` and `user_id`, you'll need to add the `user_id` query. ## 3. Add Metrics GrowthBook needs to know what metrics you want to use in your experiments, and how to query this data from BigQuery. This step depends on your needs as an organization. Keep in mind that metric queries will be generic, returning all users who do an event, but will be joined to the experiment exposure information when determining the impact of experiment results. Metrics are added from the **Analysis** → **Metrics** menu. ![Add a metric in GrowthBook](/images/guides/metrics-addmetric.png) ![Choose RudderStack data schema](/images/guides/metrics-metricoverview.png) --- ## Setting up Segment as an event tracker import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Configuring GrowthBook to work with Segment GrowthBook supports Segment as an event source for easily adding A/B testing to your application. This document is a how-to guide for setting up Segment with your application, connecting Segment to BigQuery and then connecting GrowthBook to this data. ## 1. Setup Segment If you have already set up Segment, you can skip this step. Segment allows you to send events from your application to any data destination, and supports most data warehouse. ## 2. Add Exposure Tracking Event To track when a user is exposed to an experiment from the client side, you'll want to add a Segment event call to your `trackingCallback` function. This function is called when a user is exposed to an experiment, and is passed the experiment name and variation name. ```js const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", trackingCallback: (experiment, result) => { // Example using Segment analytics.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, }); } }); ``` ## 3. Connect GrowthBook to Your Warehouse Now that we have data flowing into a data warehouse, we need to have GrowthBook connect to it. It is best practice to create another service account for GrowthBook to connect to your data warehouse. ![Choose Segment data schema](/images/new-data-sources-modal.png) After choosing Segment you'll be presented with a connection window to connect to your data warehouses. Enter the connection credentials for your data warehouse. When successfully connected, you'll be redirected to the data source page. GrowthBook will automatically create the experiment exposure query which is need to determine which user saw which experiment variation. Depending on your needs, you may still need to adjust these queries - for instance if you're tracking both `anonymous_id` and `user_id`, you'll need to add the `user_id` query. ## 3. Add Metrics GrowthBook needs to know what metrics you want to use in your experiments, and how to query this data from BigQuery. This step depends on your needs as an organization. Keep in mind that metric queries will be generic, returning all users who do an event, but will be joined to the experiment exposure information when determining the impact of experiment results. Metrics are added from the Analysis → Metrics menu. ![Edit metrics modal](/images/guides/metrics-addmetric.png) ![Metrics overview in GrowthBook](/images/guides/metrics-metricoverview.png) --- ## Setting Up Snowplow as an Event Tracker Snowplow is an advanced event tracking platform that gives you full control over your data. You can integrate it with GrowthBook to track experiment exposure events and other events. ## Prerequisites Before you begin, ensure that you have a working Snowplow tracker running on your site. Follow the [Snowplow Web Tracker Quick Start Guide](https://docs.snowplow.io/docs/sources/trackers/javascript-trackers/web-tracker/quick-start-guide/) to get set up. ## Targeting attributes Use Snowplow's `domain_user_id` as a [targeting attribute](/features/targeting) by pulling it from the Snowplow tracker and passing it to GrowthBook. **Example:** ```js window.snowplow(function() { growthbook.updateAttributes({ domain_user_id: this.sp.getDomainUserId(), }); }); ``` Alternatively, if you're accessing `sp` directly (e.g., when using the `npm` package): ```js const domainUserId = sp.getDomainUserId(); growthbook.updateAttributes({ domain_user_id: domainUserId, }); ``` ## Tracking Experiment Exposure with Snowplow To log when a user is exposed to an experiment, define a custom `trackingCallback` function in your GrowthBook SDK snippet. This sends a self-describing event to Snowplow. **Example:** ```js // In your GrowthBook SDK snippet... trackingCallback: (experiment, result) => { if (window.snowplow) { window.snowplow("trackSelfDescribingEvent", { event: { schema: "iglu:io.growthbook/experiment_viewed/jsonschema/1-0-0", data: { experimentId: experiment.key, variationId: result.key, hashAttribute: result.hashAttribute, hashValue: result.hashValue, custom: [ { key: "custom_key", value: "custom_value", }, ], }, }, }); } }; ``` Alternatively, if you're accessing `sp` directly (e.g., when using the `npm` package): ```js import { trackSelfDescribingEvent } from "@snowplow/browser-tracker"; // In your GrowthBook SDK snippet... trackingCallback: (experiment, result) => { trackSelfDescribingEvent({ event: { schema: "iglu:io.growthbook/experiment_viewed/jsonschema/1-0-0", data: { experimentId: experiment.key, variationId: result.key, hashAttribute: result.hashAttribute, hashValue: result.hashValue, custom: [ { key: "custom_key", value: "custom_value", }, ], }, }, }); }; ``` ### GrowthBook's Official Schema GrowthBook provides an [official Iglu schema](https://iglucentral.com/schemas/io.growthbook/experiment_viewed/jsonschema/1-0-0) for experiment exposure events: Schema URI: ```bash iglu:io.growthbook/experiment_viewed/jsonschema/1-0-0 ``` ## Testing your Integration with Snowplow Micro [Snowplow Micro](https://docs.snowplow.io/docs/data-product-studio/data-quality/snowplow-micro/basic-usage/) is a lightweight tool that lets you test and validate your event tracking locally. **Steps:** 1. Start Snowplow Micro: ```bash docker run -p 9090:9090 snowplow/snowplow-micro ``` 2. Point your Snowplow tracker to the local endpoint: ```js window.snowplow('newTracker', 'sp1', 'http://localhost:9090', { appId: 'my-app-id' }) ``` or... ```js newTracker('sp1', 'http://localhost:9090', { appId: 'my-app-id' }) ``` 3. Run your app with GrowthBook and Snowplow configured. When an experiment is triggered, the exposure event will be sent to Snowplow Micro. 4. Open the Snowplow Micro UI in your browser: ```bash http://localhost:9090/micro/ui ``` 5. Look for the `experiment_viewed` event to confirm the integration is working. ![Snowplow Micro UI](/images/snowplow-micro-event.png) --- ## Choose Your Data Path # Choose Your Data Path GrowthBook connects to a data warehouse to query your experiment data and calculate results. You have three options: - **Managed Warehouse** — we host a ClickHouse warehouse for you and provide built-in event tracking. - **Event Forwarder** — you keep your own warehouse, and GrowthBook streams feature and experiment exposure events (plus any custom events you send) into it for you. - **Bring Your Own Warehouse (BYOW)** — you connect your existing warehouse with your own event pipeline already feeding it. ## Quick decision **Do you already have a data warehouse?** - **No** → [Managed Warehouse](/app/managed-warehouse) gets you running quickly with zero infrastructure setup. - **Yes — and I already have feature/experiment events flowing into it** → [Bring Your Own Warehouse](/warehouses) lets you keep your existing pipelines. GrowthBook queries your data directly. - **Yes — but I don't want to build or maintain the event pipeline myself** → [Event Forwarder](/app/event-forwarder) writes feature exposure, experiment exposure, and custom events into dedicated tables in your warehouse for you. _(Requires a Pro or Enterprise plan and incurs usage fees. Currently available for BigQuery and Snowflake, with Databricks coming soon.)_ **Not sure?** Read the comparison below. ## Comparison | | Managed Warehouse | Event Forwarder | Bring Your Own Warehouse | | ------------------------ | --------------------------------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | **Setup time** | Minutes. One click to provision. | Minutes once your warehouse is connected. | Varies. Depends on your warehouse and existing pipeline. | | **Infrastructure** | Fully managed by GrowthBook (ClickHouse). | You own the warehouse; GrowthBook manages the ingestion pipeline and writes destination tables. | You manage both the warehouse and the event pipeline. | | **Event tracking** | Built-in via SDKs or Ingestion API. | Built-in via SDKs or Ingestion API; events land in your warehouse. | Use your existing event pipeline. | | **Data ownership** | Stored in GrowthBook-hosted ClickHouse. | Stays in your warehouse. | Stays in your warehouse. | | **Supported warehouses** | ClickHouse (managed). | BigQuery, Snowflake (Databricks coming soon). | BigQuery, Snowflake, Redshift, Databricks, Postgres, ClickHouse, Athena, and more. | | **Plan** | Available on all plans. | Pro or Enterprise. | Available on all plans. | | **Cost** | Included events by plan + overage pricing. | Plan price + usage-based ingestion fees + your warehouse storage costs. | Free in GrowthBook; you pay your warehouse costs. | | **Best for** | Teams getting started, no existing warehouse. | Teams that own a warehouse but don't want to build/maintain an event pipeline. | Teams with existing event pipelines and analytics infrastructure. | ## Choose Managed Warehouse if... - You don't have a data warehouse set up yet. - You want the fastest path from signup to your first experiment result. - You're evaluating GrowthBook and want to see the full experience without infrastructure setup. - Your team is small and you'd rather not manage warehouse infrastructure. - You want built-in event tracking with automatic enrichment (geo, device, UTM data). [Get started with Managed Warehouse →](/app/managed-warehouse) ## Choose Event Forwarder if... - You already own a data warehouse (BigQuery or Snowflake today; Databricks soon) but don't have a production-grade event pipeline writing exposures into it. - You want feature usage and experiment exposure events captured automatically from GrowthBook SDKs, with the option to send any other custom events via SDK helpers or our Ingestion API. - You'd like to keep your data in your own warehouse so it can be joined with the rest of your business data. - You're on (or willing to upgrade to) the Pro or Enterprise plan and are comfortable with usage-based ingestion pricing. How it works at a glance: GrowthBook provisions destination tables in your warehouse (`feature_usage`, `experiment_views`, and a general `events` table). SDK plugins automatically forward feature/experiment events; everything else goes through our Ingestion API. Events are enriched and streamed into your warehouse in near real-time. [Learn more about Event Forwarder →](/app/event-forwarder) ## Choose Bring Your Own Warehouse if... - You already have event data flowing into BigQuery, Snowflake, Databricks, Redshift, or another warehouse. - You need to join experiment data with other business data (revenue, CRM, product usage). - Your organization has data governance requirements about where data is stored. - You want full control over your data pipeline and retention policies. - You're self-hosting GrowthBook. [Connect your warehouse →](/warehouses) ## Can I switch later? Yes. You can add additional data sources (Managed Warehouse, Event Forwarder, or BYOW) at any time, and you can use more than one simultaneously. For example, you might start on Managed Warehouse for quick prototyping, then move production experiments to your own warehouse via Event Forwarder once you're ready. ## What's next - [Managed Warehouse setup guide](/app/managed-warehouse) — provision quickly and send your first event - [Event Forwarder](/app/event-forwarder) — connect your warehouse and let GrowthBook handle event delivery - [Connecting to your data warehouse](/warehouses) — step-by-step guides for each supported warehouse - [Metrics and Fact Tables](/app/metrics) — define what you want to measure in your experiments --- ## Cluster Experiments import MaxWidthImage from "@site/src/components/MaxWidthImage" # Cluster Experiments Cluster experiments are crucial when you need to **randomize at a group level** but **analyze at an individual level**. For example: - A B2B software company might want to randomize at the business level so that everyone at the company gets the same experience, but analyze individual user behavior to measure impact. - An education technology company might randomize at the school district level due to geographic constraints, but analyze outcomes at the student level. In these cases, a cluster of units nests another set of units. ![Diagram illustrating cluster-level assignment and user-level analysis in a cluster experiment. Organizations are grouped into treatment (purple) and control (teal) clusters. Each organization (e.g., Organization A, B, C, D) nests multiple users (e.g., User A1, A2, A3). Arrows show the hierarchical relationship between clusters (organizations) and individual users, with assignment occurring at the cluster level and analysis performed at the user level.](/images/cluster-experiments/cluster-diagram.png) To provide a concrete example, we will use Organizations that nest Users where your goal is to randomize at the Organization level but analyze User level behavior. ## How to run a Cluster Experiment Running a cluster experiment at the organization level while analyzing at the user level is possible in GrowthBook using Fact Tables and the robust statistics engine. Here’s how: ### 0. Prerequisites Before setting up your cluster experiment, ensure you have the following: - `organization_id` must be set up as an Identifier Type on your **Data Source** page. - If GrowthBook is handling randomization, create an Attribute `organization_id`. ### 1. Set up a Fact Table with both identifier types You need a version of your user-level metrics that has the distinct count of users as a custom denominator. To do this, create a [Fact Table](/app/metrics#fact-tables) that has both `organization_id` and `user_id` as columns, along with whatever other columns you need to create your metrics of interest. You can also use multiple fact tables with this configuration if you prefer. :::warning Avoid multiple organizations per user There must be a one-to-many relationship between `organization_id` and `user_id`. Avoid users belonging to multiple organizations, as this can introduce bias. ::: ![Fact Table example with organization and user ids, along with timestamp, event, and value columns](/images/cluster-experiments/cluster-fact-table.png) The screenshot above shows: - `user_id` and `organization_id` identifiers. - An `event` column that represents the event you are interested in measuring. - A numeric `value` column that represents the value of the event. :::info Ensure every user has at least one event Every `user_id` exposed to the experiment should have at least 1 event logged to ensure inclusion in the metric denominator. ::: ### 2. Set up your Metrics Using this Fact Table, create **Ratio Metrics** that all have the `COUNT DISTINCT` of `user_id` as their denominators. In the following example, we have a User Revenue metric: - The numerator is sum the value of all purchases across all users. - The denominator is the `COUNT DISTINCT` number of `user_id`. Note that we are not filtering the denominator to just users who made a purchase—instead, our denominator includes all users that had any event in this period. ![An example ratio fact metric with the sum of purchase values in the numerator and the distinct count of user ids in the denominator.](/images/cluster-experiments/cluster-metric.png) You can also create proportion-style ratio metrics: - The numerator is a `COUNT DISTINCT` number of `user_id` that did some particular event. - The denominator is a `COUNT DISTINCT` of `user_id` that logged any event. These metrics are aggregated by each variation, and our statistics engine uses the [Delta Method](/statistics/details#ratio-metrics) to appropriately account for the fact that you are randomizing at a different level than you are analyzing. ### 3. Create your Experiment at the Organization level All that is left to do is create your experiment where you randomize at the `organization_id` level. The following screenshot shows an example of that. ![Setting organization_id as the hash attribute.](/images/cluster-experiments/cluster-hash.png) Be sure you pick the Experiment Assignment Table that has your `organization_id` mapping. If you do not have one, you may need to set it up in your Data Source to ensure that GrowthBook knows the mapping between `organization_id` and experiment and variation assignments. ![Configuring the Experiment Assignment Table](/images/cluster-experiments/cluster-query.png) Add the metrics you created and analyze as usual! --- ## Connecting to your data warehouse # Connecting to your data warehouse :::tip Don't have a data warehouse? Try our [Managed Warehouse](/app/managed-warehouse). One-click setup, built-in event tracking, and auto-created metrics. No infrastructure setup required. ::: ## Overview One of the major benefits of GrowthBook is that your data remains securely stored in your data warehouse, and only the aggregated statistics are transmitted to GrowthBook servers or your self-hosted environment. ## Preparing your data warehouse Your data should be safe from modification as GrowthBook only runs `SELECT` queries (or the equivalent for non-SQL data sources). Still we still always recommend creating read-only users with as few permissions as possible - ideally just read permissions on the tables with the data that needs to be aggregated. If you are using GrowthBook Cloud (https://app.growthbook.io), make sure to whitelist the ip address `52.70.79.40` if applicable. ## Saving your connection configuration on Growthbook Cloud or your self hosted server The data source connection info is encrypted twice - once within the app and again by the database when persisting to disk. Most data sources have straight forward connection parameters like host, port, username, and password. However some are slightly more involved and is covered in the guides below. ## Connection guides - [AWS Athena](/warehouses/athena) - [BigQuery](/guide/bigquery) - [ClickHouse](/warehouses/clickhouse) - [Databricks](/warehouses/databricks) - [Mixpanel](/warehouses/mixpanel) - [MsSQL/SQL Server](/warehouses/ms-sql-or-sql-server) - [MySQL/MariaDB](/warehouses/mysql-or-mariadb) - [Postgres](/warehouses/postgres) - [Vertica](/warehouses/vertica) - [PrestoDB or Trino](/warehouses/prestodb-or-trino) - [Redshift](/warehouses/redshift) - [Snowflake](/warehouses/snowflake) --- ## Data Pipeline import CommercialFeature from '@site/src/components/CommercialFeature'; # Data Pipeline Mode Pipeline mode will provide substantial savings for experimenters with lots of data and/or long-running experiments. With **Pipeline Mode**, GrowthBook writes intermediate tables back to your warehouse and re-uses those across metric analyses in an experiment. Pipeline mode supports two strategies: - **Incremental refresh (Recommended)** — On each update, only scans new rows in your experiment assignment and metric sources, and persists reusable intermediate tables across updates to minimize re-scans and speed up analyses. - **Ephemeral (Legacy)** — Per-analysis, short-lived tables are created with limited retention for reuse within an analysis window. ## Incremental refresh (Recommended, in Beta) Our main pipeline mode reduces cost and speeds up experiment updates by scanning only new data since the last run, while keeping a small set of reusable, de-duplicated tables in your warehouse. Incremental refresh is only available for BigQuery, Snowflake, and Presto/Trino. Please reach out to support if your warehouse is not supported. ### How it works On each experiment update, GrowthBook only scans new data in your warehouse to keep the amount of data consumed to a minimum. Then, GrowthBook materializes several tables in your warehouse to ensure old data does not need to be scanned again, and to reduce the computation needed in future analyses. Specifically, for the experiment assignment query and fact tables, GrowthBook uses the max timestamp seen in the previous update and only scans for data since that last update, inserting that data into materialized intermediate tables to be used for analysis. ![A flow diagram illustrating an incremental data refresh pipeline for experimentation, organized into three interconnected tracks: Units, Metrics, and CUPED. New data inputs on the left—such as recent experiment exposures and metric facts—merge with existing historical data to update intermediate materialized tables in the center. Data flows sequentially from the Units table to the Metrics tables and then to the CUPED tables. Finally, the processed data from all three stages feeds into a single "Statistics" output on the right to compute analysis results.](/images/incremental-refresh-diagram.jpg) ### Main benefits - Dramatic reduction in the number of rows scanned per update, as GrowthBook only has to scan your experiment assignment queries and fact tables for data in partitions that are on or after the last timestamp seen in the previous update. - Dramatic reduction in the cost and waiting time when running on-demand dimension analyses, as GrowthBook can use the existing materialized tables, `Experiment Units Table` and one or more `Units-Date Metrics Table` to get dimension breakdowns rather than re-scanning data in your warehouse. ### Requirements - A dedicated schema/dataset with write access for GrowthBook. - Ability to retain intermediate tables beyond 24 hours (often days) for reuse. - Appropriate permissions to create/read/update (and when supported, merge/partition) tables. - Experiment assignment queries and fact tables partitioned on the `timestamp` column. If you have a different partitioning strategy, please reach out to support. To see specific instructions for your warehouse, please see the [Configuration](#configuration) section below. ### Controls and observability - Default "Update" performs an incremental refresh - "Full refresh" is available when you need to re-state results (e.g., backfills, config changes) ### Limitations and notes - While in Beta, the following analysis tools aren't supported: activation metrics and dimension breakdowns in Experiment Dashboards. - Event-level quantile metrics are supported for BigQuery and Snowflake only and rely on approximate inference techniques. Small effects may be undetectable and low sample sizes may not produce reliable results. - Changes to analysis settings sometimes require a full data refresh to take effect. In the future, we will make improvements to ensure that only the data that need be restated is restated. - Any changes to fact table sources require using a full refresh to take effect, but we are working on improvements to ensure that only the data that need be restated is restated. - Exact table schemas and partitioning/retention strategies vary by warehouse. ### Enable Incremental refresh 1. In your Data Source, open "Data Pipeline Settings" 2. Enable Pipeline Mode and select the strategy: "Incremental refresh (Recommended)" 3. Set a destination schema/dataset for intermediate tables and choose retention settings 4. Save You can test incremental refresh by enabling it only for a single experiment in the Data Pipeline Settings UI. We do not apply it to old experiments in the case that they have incompatible settings. To confirm pipeline mode is being used, you can look at the queries run for your experiment and validate that they are creating intermediate tables. ### Additional technical details On each incremental update, GrowthBook: 1. Scans the experiment assignment source for any new rows that have a timestamp greater than the max timestamp seen on the previous run and updates a de-duplicated units table (one row per unit with timestamps and experiment dimensions). 2. Scans each metric's fact source for any new rows that have a timestamp greater than the max timestamp seen on the previous run and updates pre-aggregated metric tables (one row per unit-date-refresh with timestamps and metric dimensions). 3. Optionally loads pre-experiment windows for CUPED into a separate table for only newly seen units. 4. Joins the above to produce summary statistics and per-dimension breakdowns. Therefore, GrowthBook will materialize the following tables per experiment: - The units table (`Experiment Units Table` - named `gb_units_` in your warehouse) - one row per unit in the experiment with the unit id, variation id, first exposure timestamp, and experiment dimensions - this table is joined to new exposures, de-duped, and then re-created on each update - One or more metrics tables depending on the number of metric sources and the number of metrics (`Unit-Date Metrics Table` - named `gb_metrics___` in your warehouse) - one row per unit-date-refresh, as we take new metric values and aggregate them by unit and date on each refresh and insert them into the metrics table - we insert new rows into this table on each update - An additional table per above metric source for CUPED (`CUPED Metrics Table` - named `gb_cuped____covariate` in your warehouse) - one row per unit, as we scan the metric source for the pre-experiment window, join it to only new units found in the units update, and then insert the aggregated values per unit into the CUPED table - we insert new rows into this table on each update On each update, we look for new rows in your sources. We define "new" rows as those where the row has a timestamp greater than the max timestamp seen on the previous run. This means that if any data is re-stated in your warehouse, or new rows land that are linked to old timestamps, GrowthBook may miss this data unless you do a full refresh. In other words, as data lands, it must have a later timestamp than the data as of the last update or it will not be included in the analysis. This also means that if a Fact Table or Experiment Assignment Table definition includes references to multiple source tables, these tables should be updated at the same time to ensure GrowthBook can capture all data. GrowthBook is building functionality to support additional use cases, improve control over intermediate table retention, and support additional data sources. Please reach out to support if you have any concerns or questions. ## Configuration :::info Pipeline mode requires write permissions While Pipeline mode requires you to give write permissions to your Data Source, GrowthBook only uses it for the Pipeline Mode. All other user-provided SQL statements are validated to be read-only before being executed. ::: ### BigQuery 1. (strongly recommended, but optional) Create a dedicated dataset to which GrowthBook will write tables. This will keep your data warehouse clean and ensure that we are only writing to a dedicated space. 2. Grant permissions to create tables to the role connecting GrowthBook to your warehouse. You can do this by granting your GrowthBook Service Account the `BigQuery Data Editor` role on the new dataset. You can also give only BigQuery table reading and writing permissions on that dataset if you want to be more restrictive. 3. Navigate to your BigQuery Data Source in GrowthBook and scroll down to "Data Pipeline Settings" 4. Click "Edit" and enable pipeline mode, set the destination dataset to your new dedicated GrowthBook dataset from step 1. ### Snowflake 1. (strongly recommended, but optional) Create a dedicated schema to which GrowthBook will write tables. This will keep your data warehouse clean and ensure that we are only writing to a dedicated space. 2. Grant permissions to create tables to the role connecting GrowthBook to your warehouse. The Snowflake role attached to GrowthBook will need `CREATE TABLE`, `SELECT - FUTURE TABLE`, and `USAGE` on the schema created in step 1. For Inremental Refresh, it may also need `OWNERSHIP - FUTURE TABLE` and `MODIFY` permissions on that dedicated schema. 3. Navigate to your Snowflake Data Source in GrowthBook and scroll down to "Data Pipeline Settings" 4. Click "Edit" and enable pipeline mode, set the destination schema to your new dedicated GrowthBook schema from step 1. ### Databricks (Ephemeral only) Databricks works slightly differently. Instead of creating a temporary table, we create a regular table for the deduped units assignment and then `DROP` that table when analysis is completed. :::info Using pipeline mode in Databricks requires DROP permissions Using pipeline mode in Databricks requires either granting DROP permissions to the Databricks account that GrowthBook uses, or leaving many tables in your schema you have to manually delete later! For this reason we strongly recommend a standalone schema for GrowthBook to use to write tables to. ::: 1. (strongly recommended, but optional) Create a dedicated schema to which GrowthBook will write temporary tables. This will keep your data warehouse clean and ensure that we are only writing to and dropping from a dedicated space. 2. Grant permissions to your user account or service principal that already has read permission in your warehouse. That user/service principle will need to be able to `USE SCHEMA`, `CREATE TABLE`, `DROP TABLE`, and to `SELECT` and `EXECUTE` in the schema. 3. Navigate to your Databricks Data Source in GrowthBook and scroll down to "Data Pipeline Settings" 4. Click "Edit" and enable pipeline mode, set the destination schema to your new dedicated GrowthBook schema from step 1, and whether you want the table to be deleted (we recommend you leave this setting on as we will not re-use these tables at a later date). If this setting is off, you'll need to manually delete the tables that GrowthBook creates. --- ## Data Source Configuration import Anchor from '@site/src/components/Anchor' # Data Source Configuration ## Overview Data Sources are how GrowthBook connects to your data warehouse so that it can pull those aggregated statistics in order to compute metrics and experiment results. Each Data Source defines how to connect to your data, what version of SQL to use when querying your data, and can provide templates for what the SQL to connect to your Data Source should look like depending upon which event tracker software you use. GrowthBook works with your existing SQL data, no matter where it is located and no matter what shape or format it is in, whether you have a strongly normalized schema, a single “events” table with JSON fields, or something in between. ## Supported Event Schemas When adding a new Data Source in Growthbook from /datasources page we first guide you to select what event tracker software you use, or you can choose custom if you don't use any of the popular third party event trackers that we support. Telling us which event tracker you use, gives us an idea on the likely shape of your data. This will help us generate the correct sql to extract out the aggregated statistics from your site with as little modification on your end as possible. Here is Growthbook's current list of event trackers that we support with links for more details on how to set them up with GrowthBook: - [Amplitude](/event-trackers/amplitude) - [CleverTap](/event-trackers/clevertap) - [Firebase](/event-trackers/firebase) - [Freshpaint](/event-trackers/freshpaint) - [Fullstory](/event-trackers/fullstory) - [Google Analytics 4 (BigQuery only)](/guide/GA4-google-analytics) - [Heap Analytics](/event-trackers/heap) - [Jitsu](/event-trackers/jitsu) - [Keen IO](/event-trackers/keenio) - [Matomo](/guide/matomo) - [Mixpanel](/guide/mixpanel) - [MParticle](/event-trackers/mparticle) - [RudderStack](/guide/rudderstack) - [Segment](/event-trackers/segment) - [Snowplow](/event-trackers/snowplow) If you do not use any of those you choose [Custom Data Source](/event-trackers/custom) and define some of the sql yourself. ## Configuration Settings Once you have chosen your event tracker and data source type and successfully connected, you will be given an opportunity to modify your configuration settings. For many applications Growthbook will have chosen the correct configuration settings straight out of the box based upon which event tracker you choose. In some instances, you may need to tweak them slightly, or in the case of using a custom datasource, define them more explicitly. ### Identifier Types These are all of the types of identifiers you use to split traffic in an experiment and track metric conversions. Common examples are `user_id`, `anonymous_id`, `device_id`, and `ip_address`. There are some cases where a single database column isn't enough to uniquely identify a subject. For example, you might need the combination of `company` and `user_id`. In this case, we recommend creating a synthetic identifier by concatenating all of the fields together. For example, you can create a `company_user` identifier and then in your SQL, select it as follows: `CONCAT(company, user_id) as company_user`. ### Experiment Assignment Queries An experiment assignment query returns which users were part of which experiment, what variation they saw, and when they saw it. Each assignment query is tied to a single identifier type (defined above). You can also have multiple assignment queries if you store that data in different tables, for example, one from your email system and one from your back-end. :::note Queries used to generate experiment results Assignment queries are one-half of the queries that are used to generate experiment results, the other being metric queries. Assignment queries can be edited from the `Metrics and Data` → `Data Sources` page. ::: The end result of the query should return data like this: | user_id | timestamp | experiment_id | variation_id | | ------- | ------------------- | -------------- | ------------ | | 123 | 2021-08-23-10:53:04 | my-button-test | 0 | | 456 | 2021-08-23 10:53:06 | my-button-test | 1 | The above assumes the identifier type you are using is `user_id`. If you are using a different identifier, you would use a different column name. Here's an example query you might use: ```sql SELECT user_id, received_at as timestamp, experiment_id, variation_id FROM events WHERE event_type = 'viewed experiment' ``` Make sure to return the exact column names that GrowthBook is expecting. If your table’s columns use a different name, add an alias in the SELECT list (e.g. `SELECT original_column as new_column`). #### Duplicate Rows If a user sees an experiment multiple times, you should return multiple rows in your assignment query, one for each time the user was exposed to the experiment. This helps us detect when users were exposed to more than one variation, and eventually may be useful in helping build interesting time series. #### Experiment Dimensions In addition to the standard 4 columns above, you can also select additional dimension columns. For example, `browser` or `referrer`. These extra columns can be used to drill down into experiment results. #### Identifier Join Tables If you have multiple identifier types and want to be able to auto-merge them together during analysis, you also need to define identifier join tables. For example, if your experiment is assigned based on `device_id`, but the conversion metric only has a `user_id` column. These queries are very simple and just need to return columns for each of the identifier types being joined. For example: ```sql SELECT user_id, device_id FROM logins ``` #### SQL Templates GrowthBook uses [Handlebars](https://handlebarsjs.com/guide/#language-features) to compile our SQL queries and we expose several variables and helper functions to make your SQL dynamic and reusable. This applies to experiment assignment queries, fact tables, and metrics. Read more about [SQL Templates](/app/sql-templates) and the different variables that are available. ### Jupyter Notebook Query Runner This setting is only required if you want to export experiment results as a Jupyter Notebook. There is no one standard way to store credentials or run SQL queries from Jupyter notebooks, so GrowthBook lets you define your own Python function. It needs to be called `runQuery`, accept a single string argument named `sql`, and return a pandas data frame. Here's an example for a Postgres (or Redshift) data source: ```python import os import psycopg2 import pandas as pd from sqlalchemy import create_engine, text # Use environment variables or similar for passwords! password = os.getenv('POSTGRES_PW') connStr = f'postgresql+psycopg2://user:{password}@localhost' dbConnection = create_engine(connStr).connect(); def runQuery(sql): return pd.read_sql(text(sql), dbConnection) ``` **Note:** This python source is stored as plain text in the database. Do not hard-code passwords or sensitive info. Use environment variables (shown above) or another credential store instead. ## Schema Browser When you connect a supported data source to GrowthBook, we automatically generate metadata that is used by our Schema Browser. The Schema Browser is a user-friendly interface that makes writing queries easier as you can easily explore information about the datasource such as databases, schemas, tables, columns, and data types. ![GrowthBook Schema Browser](/images/growthbook-schema-browser.png) Below are the data sources that currently support the Schema Browser: - AWS Athena - _Requires a Default Catalog_ - BigQuery - _Requires a Project Name and Default Dataset_ - ClickHouse - Databricks - _Currently only supported on version 10.2 and above with a Unity Catalog_ - MsSQL/SQL Server - MySQL/MariaDB - Postgres - PrestoDB (and Trino) - _Requires a Default Catalog_ - Redshift - Snowflake :::note Generating the schema manually If you added a supported data source prior to GrowthBook v2.0, you can generate the schema manually by clicking "Data Sources" on the left-nav, selecting the data source, and then clicking the "View Schema Browser" button and following the on-screen prompt. ::: --- ## Experiment Decision Framework import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; # Experiment Decision Framework The Experiment Decision Framework (EDF) is a set of tools and customizable settings to help streamline and improve experiment decision making. The EDF can help answer questions like, "Should I keep running my experiment?" and "Do my results meet my success criteria?" The EDF has two key components: 1. The Decision Criteria 2. The Target Minimum Detectable Effects (MDEs) for your goal metrics ### Decision Criteria The **Decision Criteria** are the ways that your results in an experiment lead to a decision. Do all goal metrics need to be statistically significant and positive to ship, or are you happy to ship so long as none are statistically significant and negative? These can be customized or you can use our two out-of-the-box options: (1) Clear Signals, which requires no guardrail failures and all goal metric successes, or (2) Do No Harm, which only requires that no metrics are statistically significant in the harmful direction. ![Example Decision Criteria](/images/experiment-decision-framework/decision-criteria.png) ### Target Minimum Detectable Effects The **Target Minimum Detectable Effects** (MDEs) are the smallest effects you want to be able to reliably detect before making a decision. For example, suppose 10% is your target MDE value (10% is the GrowthBook default). If your confidence intervals range from 10 to 20\%, then you have reached your desired precision. Once this desired precision is reached, your experiment has collected enough data for the decision criteria to be applied and a ship, rollback, or review decision is given for your experiment. Note: sometimes a decision is given before the target MDE is reached if either (1) results are significant at a very strict threshold or (2) if you have sequential testing enabled. Once an experiment has reached the needed precision, we make a recommendation based on the Decision Criteria for your experiment. ![Example Rendered Decision](/images/experiment-decision-framework/experiment-decision.png) ## Setting up the Experiment Decision Framework You can set up the Experiment Decision Framework under **Settings > General > Experiment Settings**. ![Experiment Decision Framework Organization Settings](/images/experiment-decision-framework/organization-settings.png) 1. Under `Settings > General > Experiment Settings`, set the `Minimum experiment runtime`, which stops the decision framework from displaying days remaining or experiment decisions while early data collection is ongoing. The default value is 3 days, but larger values such as 7 may be reasonable, especially if there are strong day of the week effects in your data. 2. Set the `Target Minimum Detectable Effects` (target MDEs) for your key goal metrics. You can set the default for all metrics on the `Settings > General > Metric & Data` tab. The GrowthBook default is 10%, with [selection guidance](#selecting-target-minimum-detectable-effects) below. 3. Create the `Decision Criteria` that you want your experiments to follow under `Settings > General > Experiment Settings`. 4. Review your Experiment creation process! The EDF relies heavily upon using your Goal Metrics to determine how long your experiment needs to run as well as what decision to make. If you have 3 or more goal metrics, or even 2 competing goal metrics, it can take a lot longer for your experiment to finish and reach a clear shipping recommendation. Consider setting only one goal metric, and use secondary metrics to deep dive your results and build more understanding. ### Customizing the decision framework per experiment The above process lets you select defaults for your Organization, but the decision framework can also be customized per experiment. On the Experiment Overview Tab, under Analysis Settings, you can customize the target MDEs for the goal metrics in your experiment or change the decision criteria to apply to the experiment. ![Experiment Decision Framework Settings](/images/experiment-decision-framework/experiment-decision-framework-settings.png) ### Selecting Target Minimum Detectable Effects The target MDE is the smallest effect you want to be able to reliably detect before making a decision. For example, suppose 10% is your target MDE value (10% is the GrowthBook default). If your confidence intervals range from 0 to 20\%, then a 10\% lift would be statistically significant, your target MDE has been reached, and your experiment is ready for decision. Please note that the time at which you reach your target MDE is unaffected by the lift estimate in your experiment. In the above example, only the inputted target MDE and the width of your confidence interval are used to determine if you have reached your target MDE. If your estimated lift in your experiment was 5%, but your confidence interval ranged from -5\% to 15\%, your results would not be statistically significant even though you reached your target MDE. That means you're comfortable making a ship or roll back decision with this amount of power, even if your results are not statistically significant. If your estimated lift was 20\% with the same interval width, then your results would be statistically significant, and you should ship. The target MDE helps ensure that the experiments you run are well-powered, not that results from an individual experiment will be statistically significant. Your target MDE should balance the lift at which it is profitable to ship against the time needed to run the experiment. Small target MDEs will let you detect tinier effects before being ready to make a decision, but they will require more data. Large target MDEs will shorten experiment runtimes, but if your true effects are smaller than your target MDEs, you will often face ambiguity at the end of your experiment. ## Experiment Statuses Experiments have a variety of statuses represented as badges on the Experiment Detail page and in the Experiment List based on whether they are in Draft, Running, or are Stopped. For Stopped experiments, your stated reason for calling the experiment is included in the status. For running experiments, there are a variety of statuses, each depending on the current state of your experiment and your settings. Many statuses have an associated tooltip that explains them in more detail. Note: only `unhealthy` and `no data` statuses appear before your experiment has been running for your organization's `minimum experiment duration`, which defaults to 3 days. | Status | Status conditions | What to do next | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Unhealthy | This status appears when your experiment results has imbalanced traffic (SRM), multiple exposures, or is low-powered. | For SRM or multiple exposures, [see the troubleshooting doc](/kb/experiments/troubleshooting-experiments). For low-powered experiments, see the FAQ below. | | No data | This status appears when your experiment results have refreshed, but there are no users showing up in your experiment. | You have no traffic or are missing some datasource and metric configurations. There are three common causes if you have no traffic: (1) you just started your experiment and you just need to wait for your experiment exposure data to land in your data warehouse before it will show up in GrowthBook; (2) you started your experiment but there are no linked feature flags, visual editor changes, or URL redirects attached (see the Experiment Overview > Implementation section) that are actually exposing users to your experiment and getting traffic; (3) you have some issue configuring your Experiment Assignment Query or something else in your GrowthBook set-up that is likely affecting multiple experiments ([more information here](/kb/experiments/troubleshooting-experiments)). | | ~X days left | This status appears when your experiment has not yet reached the [target power](#what-is-target-power) and it estimates how much longer you need to run the experiment to reach that target. | Continue to collect data until your target power is reached, or the experiment has been running longer than is feasible for your business goals. If the estimated duration is quite high, it is likely you have: (1) too many goal metrics, (2) your target minimum detectable effect (MDE) for your goal metrics is too small for your traffic, or (3) you are not getting very much new traffic in the experiment. We recommend selecting only 1 or 2 goal metrics and consider establishing more realistic target MDEs for your key goal metrics. | | Ship now | This status appears when your decision criteria to Ship Now have been met for at least one variation AND either (1) Sequential Testing is enabled in our Frequentist engine OR (2) the goal metrics have reached their target MDEs (3) OR the effects are statistically significant at a strict threshold (Frequentist engine: p-value 99.9\%). | Ship the variation that fits the above criteria! | | Roll back now | This status appears when your decision criteria to Rollback Now have been met for all test variations AND either (1) Sequential Testing is enabled in our Frequentist engine OR (2) the goal metrics have reached their target MDEs (3) OR the effects are statistically significant at a strict threshold (Frequentist engine: p-value 99.9\%) | Stop the experiment and keep the status quo! | | Ready for review | This status appears when your decision criteria to Review have been met for any variation AND either (1) Sequential Testing is enabled in our Frequentist engine OR (2) the goal metrics have reached their target MDEs (3) OR the effects are statistically significant at a strict threshold (Frequentist engine: p-value 99.9\%) cases. | Consider the trade-offs between shipping, rolling back and iterating, or continuing to run the experiment in these ambiguous cases. | ## FAQ ### What is target power? Your experiment reaches its "target power" when it has enough data to reliably detect the target minimum detectable effect for all of your goal metrics. More data will help get more precise results, but based on the level of precision you want for your goal metrics, you have enough data to make a decision. ### What should I do if my experiment has many days remaining or low power? In these cases, the experiment traffic is not high enough to reliably detect the target MDE you have set for all of your goal metrics. This can often happen when: 1. Your target MDE is too low for your traffic and metric - if you run low traffic experiments, or if you have a conversion metric that is very rare (e.g. less than 1\%), then it can take a lot of data to precisely estimate effects. It is possible your target MDE for your metric is too low. Consider evaluating whether you are comfortable making decisions with more uncertainty, and consider increasing your target MDE in the settings for that metric. 2. You have too many goal metrics - ideally you only have one or two goal metrics that decide whether or not you will launch an experiment variation. Picking a small number of goal metrics makes it easier to get enough power to make a decision. It forces you to clearly state up front your goals for your experiment, which makes decision making more straightforward. Finally, it can protect you from making up business goals to fit your results, rather than making sure your results fit your business goals. 3. You have too many variations - having more variations splits your traffic and lowers the power you have to detect effects. --- ## Dimensions import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Dimensions Dimensions let you drill down into experiment results. For example, if you define a `browser` dimension, you can see how Safari users behaved vs Chrome. Be careful, the more dimensions and metrics you look at for an experiment, the more likely you are to see false positives - something that looks significant when it really isn't. For example, if you break out results by country, it's pretty likely that at least one of the 100+ countries in your dataset will be significantly different just by random chance. It's best to treat dimensions as an exploratory tool and not something to directly draw conclusions from. The two best use cases are identfying bugs (the `browser` example) and getting ideas for dedicated follow-up experiments. Dimensions are supported for both Mixpanel and SQL data sources. ## Breaking down by more than one dimension GrowthBook currently only supports selecting a single dimension at a time when viewing experiment results. There isn't native support yet for combining two or more dimensions into one breakdown, but there are two workarounds: - **Custom SQL filter**: If one of your dimensions already exists on your experiment assignment query, add a custom SQL filter in the experiment's analysis settings to restrict results to a specific value of that dimension, then use the dimension selector to break down by the second dimension. - **Custom dimension SQL**: Create a new dimension whose SQL computes the intersection of the two dimensions you care about — for example, concatenating `country` and `account_type` into a single value like `US - premium`. See the [User Dimensions](#user-dimensions) section below for how to write custom dimension SQL. ## SQL There are two types of dimensions for SQL data sources: Experiment Dimensions and User Dimensions. Experiment dimensions are more reliable for producing unbiased experiment analyses because we can always know the dimension that is associated with the first experiment exposure for a unit. Using dimension data that could be affected by the experiment (e.g. that is collected _after_ the first experiment exposure) could bias dimension drill-downs. Therefore, use user dimensions with caution, and we suggest using immutable dimensions as your user dimensions (e.g. the first client the user ever used, or the first country the user ever logged in from). For all experiment analyses, we only ever choose one dimension per user, to avoid the aforementioned issues with potential bias. For Experiment Dimensions, we select the dimension associated with the user's first exposure event for the experiment. For User Dimensions, we strongly suggest only having one dimension value per user, as we cannot discern what dimension a user had before the experiment. Instead, we simply choose the `MAX(value)` for the user dimension. ### Experiment Dimensions These are attributes that are specific to the point-in-time that a user was put into an experiment. For example, `browser` or `referrer`. Instead of a standalone SQL query, experiment dimensions are simply extra columns you return from the Experiment assignment query defined for your data source. As an example, if you set the following as your experiment assignment query: ```sql SELECT user_id, received_at as timestamp, experiment_id, variation_id, browser FROM experiment_viewed ``` The first 4 columns are standard, but `browser` is a custom one that can be used as an Experiment Dimension. #### Configuring Experiment Dimensions To power the health tab and pre-computed dimension results, you need to configure your Experiment Dimensions. You can do this by clicking the "Configure Dimensions" button in the kebab menu next to the exposure query where the Experiment Dimensions are defined: ![Configure Dimensions Option](/images/configure-dimensions.png) In the modal that pops up, you can run a query to find the top 20 dimension slices for your Experiment Dimensions and save them for use on the health tab. If you want to have more control over the pre-defined slices, the best way to do so is to modify the Exposure Query SQL to include a `CASE` statement that defines the dimension slices. For example, if you wanted to have a `browser` dimension with slices for `Chrome`, `Safari`, and `Other`, you could do something like this: ```sql SELECT user_id, received_at as timestamp, experiment_id, variation_id, CASE WHEN browser LIKE '%Chrome%' THEN 'Chrome' WHEN browser LIKE '%Safari%' THEN 'Safari' ELSE 'Other' END as browser FROM experiment_viewed ``` ### User Dimensions These are attributes of your users that are relatively stable over time, and ideally, do not change over the course of an experiment. For example, `cohort`, `initial age group`, or `first client used`. A user dimension is defined by a simple SQL query that returns two columns: an identifier and `value`. The name of the identifier column depends on which identifier type the dimension is using. Remember, when using these for analysis, we will pick the `MAX(value)` for each user, and therefore it is best to only have one value per user. Here's an example SQL using an immutable attribute (the user's first country): ```sql -- Assumes identifier type is "user_id" SELECT user_id, first_country as value FROM users ``` It's best to keep the number of unique values for a dimension small if possible to avoid the False Positive issues discussed above. We automatically cap the number at 20, but you can do it yourself if you want more control. Here's an example using the "first_country" dimension: ```sql SELECT user_id, ( CASE WHEN first_country = 'us' THEN 'US' WHEN first_country = 'uk' THEN 'UK' ELSE 'Other' END ) as value FROM users ``` If you need to use a dimension that can change over time, you should use the `{{ startDate }}` template variable to pin it to the value the user had before the experiment started: ```sql SELECT user_id, plan_type as value FROM subscriptions WHERE timestamp { // ...some complex logic return dimensionValue })() ``` You can also reference the experiment start/end date in your javascript expression. For example, if you add a super event called `userRegistrationDate` that stores a unix timestamp, you could make a `New vs Existing` dimension like this: ``` event.properties.userRegistrationDate >= {{ startDateUnix }} ? "new" : "existing" ``` The variables you can reference are: - **startDate** - `YYYY-MM-DD HH:mm:ss` of the earliest data that needs to be included - **startYear** - Just the `YYYY` of the startDate - **startMonth** - Just the `MM` of the startDate - **startDay** - Just the `DD` of the startDate - **startDateUnix** - Unix timestamp of the startDate (seconds since Jan 1, 1970) - **endDate** - `YYYY-MM-DD HH:mm:ss` of the latest data that needs to be included - **endYear** - Just the `YYYY` of the endDate - **endMonth** - Just the `MM` of the endDate - **endDay** - Just the `DD` of the endDate - **endDateUnix** - Unix timestamp of the endDate (seconds since Jan 1, 1970) - **experimentId** - Either a specific experiment id OR `%` if you should include all experiments --- ## Event Forwarder # Event Forwarder GrowthBook offers a fully managed event ingestion pipeline. Use our SDKs to track events in your app and we will enrich and forward them to your data warehouse in near real-time. ## Benefits - **Fully Managed**: No need to worry about infrastructure management, scaling, or maintenance. - **Seamless Integration**: Built-in tracking in our SDKs. - **Near Real-Time**: Events are available in your warehouse typically within a few seconds. - **Broad Support**: Works with BigQuery and Snowflake today, with more warehouses coming soon. - **Works with Self-Hosting** - Use with either GrowthBook Cloud or your self-hosted GrowthBook instance. ## How it Works 1. Provide GrowthBook with your data warehouse connection details (see [Connection Permissions](#connection-permissions) for required permissions) 2. We will create destination tables for `events`, `experiment_viewed`, and `feature_usage` 3. You send analytics events to our scalable Ingestion API 4. We enrich the events and stream them into your warehouse within seconds 5. We will build a Fact Table, customize the Data Source's experiment assignment queries, and feature usage queries, so you can quickly start analyzing your experiment data and defining metrics. ## Connection Permissions When setting up the Event Forwarder, GrowthBook uses your data warehouse connection to pre-create destination tables and validate write access before provisioning the Confluent connector. ### BigQuery The service account configured in your GrowthBook BigQuery datasource must have the **BigQuery Data Editor** role on the destination dataset. This grants the necessary permissions to: - Check whether destination tables exist - Create the destination tables (`events`, `experiment_viewed`, `feature_usage`) - Write data via the BigQuery Storage Write API You can grant this at the dataset level in the BigQuery console or via the `roles/bigquery.dataEditor` IAM role binding. ### Snowflake The Snowflake connection **must use key-pair authentication** — password authentication is not supported for the Event Forwarder. The Snowflake role you provide must have the following privileges: | Privilege | Object | | --------------- | ------------------------ | | `USAGE` | Database | | `USAGE` | Schema | | `USAGE` | Warehouse (if specified) | | `CREATE SCHEMA` | Database | | `CREATE TABLE` | Schema | | `INSERT` | Tables | `CREATE TABLE` and `CREATE SCHEMA` are required because the Confluent Snowflake Sink uses Snowpipe Streaming with schema evolution, which may create or alter tables and schemas as your event schema changes. Key-pair authentication must be configured on the Snowflake datasource connection itself — it is not set in the Event Forwarder UI. Update your datasource's connection settings to use key-pair auth before enabling the Event Forwarder. ## Get Started The event forwarder is an advanced Enterprise feature and must be enabled for your account. Contact your account manager or reach out to [sales@growthbook.io](mailto:sales@growthbook.io) to learn more and get started. ## Sending Events There are 2 ways to send events to GrowthBook: 1. With our [SDKs](#with-sdks) (limited language support) 2. With our [Ingestion API](#ingestion-api) ### With SDKs The following SDKs have a built-in plugin to automatically send events. - [HTML Script Tag](#html-script-tag) - [Client-Side JavaScript / React](#client-side-javascript--react) - [Node.js](#nodejs) For everything else, use the [Ingestion API](#ingestion-api). When the plugin is added, these SDKs will automatically send feature usage and experiment view events to GrowthBook. They also expose a helper method to log additional custom events with optional properties. All of the attributes in the SDK are sent along with events as context. #### HTML Script Tag Simply add `data-tracking="growthbook"` to your script tag to enable. ```html ``` To track additional events, use the `window.gbEvents` global variable. You can push events to this array, and they will be tracked. ```html // Ensure the global variable exists window.gbEvents = window.gbEvents || []; // Simple (no properties) window.gbEvents.push("Page View"); function handleSignUpClick() { // With custom properties window.gbEvents.push({ eventName: "Button Click", properties: { button: "Sign Up" } }); } Sign Up ``` #### Client-Side JavaScript / React Use the `growthbookTrackingPlugin` to enable tracking. We recommend also using the `autoAttributesPlugin` to include many common attributes in your events (browser, session_id, etc.). ```js import { GrowthBook } from "@growthbook/growthbook"; import { autoAttributesPlugin, growthbookTrackingPlugin } from "@growthbook/growthbook/plugins"; const gb = new GrowthBook({ clientKey: "YOUR_CLIENT_KEY", plugins: [ autoAttributesPlugin(), growthbookTrackingPlugin() ] }); ``` Use the `logEvent` method to track additional custom events: ```js // Simple (no properties) gb.logEvent("Page View"); // With custom properties gb.logEvent("Button Click", { button: "Sign Up", }); ``` #### Event Delivery on Page Navigation Events are batched and sent with `fetch` using `keepalive`, and when the page is hidden or unloading the queue is flushed with `navigator.sendBeacon`. Both survive the page being torn down, so exposure events are not lost when a user navigates away immediately — which matters especially for **redirect experiments**, where the losing request would otherwise skew traffic splits and trigger SRM warnings. You can override this with the `transport` option (`data-event-transport` on the script tag): `"auto"` (default, described above), `"beacon"` (always prefer `sendBeacon`), or `"fetch"` (never use `sendBeacon`). ```js growthbookTrackingPlugin({ transport: "beacon" }); ``` #### Node.js Use the `growthbookTrackingPlugin` to enable tracking: ```js import { GrowthBookClient } from "@growthbook/growthbook"; import { growthbookTrackingPlugin } from "@growthbook/growthbook/plugins"; const gb = new GrowthBookClient({ clientKey: process.env.GROWTHBOOK_CLIENT_KEY, plugins: [growthbookTrackingPlugin()], }); ``` Use the `logEvent` method to track additional custom events: ```js gb.logEvent("Sign Up", { accountPlan: "pro" }, userContext); ``` User-scoped instances also have a `logEvent` method that doesn't require the user context: ```js req.growthbook.logEvent("Sign Up", { accountPlan: "pro" }); ``` #### Sending events to the right region The tracking plugin sends events to `https://us-east-1.gb-ingest.com` by default. If you selected **eu-west-1** as your Event Forwarder's Data Region, override the ingestor host so events reach the right Kafka/Confluent resources instead of being dropped: - **HTML Script Tag**: add `data-event-ingestor-host="https://eu-west-1.gb-ingest.com"` to the script tag. - **JavaScript / React / Node.js**: pass `ingestorHost` to `growthbookTrackingPlugin()`: ```js growthbookTrackingPlugin({ ingestorHost: "https://eu-west-1.gb-ingest.com", }) ``` You can find your Event Forwarder's configured region on the data source's Event Forwarder settings. ### Ingestion API You can also send events directly to our ingestion API. Pass an array of event objects, each with the following properties: - **event_name**: The name of the event (e.g., "Purchase", "Button Click") - **properties**: Optional key-value pairs with properties of the event itself - **attributes**: Optional key-value pairs with attributes of the user or context at the time of the event - YOUR_WAREHOUSE_REGION: either us-east-1 or eu-west-1 depending on what you selected when creating your datasource - YOUR_CLIENT_KEY: The key from your sdk connection. Go to https://app.growthbook.io/sdks/ click on your connection or make a new one. Copy the text from the "Client Key" ```bash curl -X POST "https://YOUR_WAREHOUSE_REGION.gb-ingest.com/track?client_key=YOUR_CLIENT_KEY" \ -H "Content-Type: application/json" \ -d '[{ "event_name": "Purchase", "properties": { "amount": 100 }, "attributes": { "user_id": "12345" } }]' ``` Using the wrong host means events won't reach the Kafka/Confluent resources your forwarder is actually wired up to. #### Experiment View Events In order to use GrowthBook's experiment analysis features, you must send an event every time a user views an experiment. It must match the following format: - **event_name**: Must be `"Experiment Viewed"` - **properties**: Must include the following key/value pairs: - `experimentId`: The ID of the experiment being viewed - `variationId`: The ID of the variation that was shown to the user In addition, you must include `attributes` with the user attributes that were used to evaluate the experiment, plus any attributes you want to use as dimensions for slicing and dicing. #### Feature Usage Events To take advantage of GrowthBook's feature usage analytics, you must send an event every time a feature is evaluated with a specific format. - **event_name**: Must be `"Feature Evaluated"` - **properties**: Must include the following key/value pairs: - `feature`: The name of the feature being evaluated - `value`: The feature's value that was returned from the evaluation - `source`: (optional) The source of the feature value (e.g., "defaultValue", "experiment", "force") - `ruleId`: (optional) The ID of the specific rule that was used to evaluate the feature (or `$default` if the default value was used) - `variationId`: (optional) If the value came from an experiment, the ID of the variation that was returned #### Attributes Attributes are key-value pairs that provide context about the user or environment at the time of the event. They can be used to slice and dice your data in analysis. It's recommended to include the same attributes you use in your GrowthBook SDK. Some attributes are enriched in the ingestion API if provided: - `ip` - a geoip lookup is done and the following attributes are added. If an ip attribute is not provided, we will use the IP address of the request. - `geo_country` - `geo_city` - `geo_lat` - `geo_lon` - `ua` - the user agent is parsed and the following attributes are added. If a user agent attribute is not provided, we will use the user agent of the request. - `ua_browser` (e.g. Safari) - `ua_os` (e.g. macOS) - `ua_device_type` (e.g. mobile) - `url` - the URL is parsed and the following attributes are added: - `url_path` (e.g. /products/123) - `url_host` (e.g. www.example.com) - `url_query` (e.g. ?utm_source=google) - `url_fragment` (e.g. #section1) It's also recommend to include attributes about which SDK is being used. This can help with debugging. - `sdk_language` (e.g. python) - `sdk_version` (e.g. 1.2.3) #### Limits When calling the ingestion API directly, please be aware of the following default limits: - Maximum of 100 events per request - Maximum of 1 request per second These limits are to protect accidental misuse of the API. Our Ingestion API can handle much higher volumes than this. Reach out to your account manager if you need to increase your limits. --- ## Experiment Bulk Results API # Experiment Bulk Results API `GET /api/v1/experiments/{id}/bulk-results` returns every successful snapshot for one experiment generated inside a date window, flattened into `ExperimentBulkResult` items. Use it to mirror historical results into a warehouse or notebook; use `GET /experiments/{id}/results` when you only need the latest analysis. :::info Off by default and excluded from the [REST API reference](/api). Self-hosted deployments opt in with `EXPERIMENT_BULK_RESULTS_ENABLED=true`. While disabled, every request answers `404 {"message": "Unknown API endpoint"}` — indistinguishable from an endpoint that does not exist. This choice is because this endpoint can possibly consume a lot of resources from the back-end deployment if the requested payload is enormous, so it should be used with caution. ::: ## Request Authenticate with a Personal Access Token or secret key that can read the experiment's Project. ```bash curl -X GET 'https://api.growthbook.io/api/v1/experiments/exp_abc123/bulk-results?dateStart=2026-01-01T00:00:00Z&dateEnd=2026-02-01T00:00:00Z' \ -H 'Authorization: Bearer YOUR_API_KEY' ``` | Param | Required | Description | | -------------- | -------- | ------------------------------------------------------------------ | | `dateStart` | yes | ISO 8601 date-time. Snapshots generated on or after this instant. | | `dateEnd` | yes | ISO 8601 date-time. Snapshots generated on or before this instant. | | `phase` | no | Zero-based phase index. Omit for all phases. | | `snapshotType` | no | `standard`, `exploratory`, or `report`. Omit for all types. | | `limit` | no | Snapshots per page, 1–100. Defaults to 10. | | `offset` | no | Snapshots to skip. Defaults to 0. | The query is strict: an unrecognized param is a `400`. Other failures are `400` for a malformed `dateStart`/`dateEnd`/`phase` or an out-of-range `limit`, `404` for an unknown or unreadable experiment id, and `429` past the per-key cap (60 requests/minute, set by `EXPERIMENT_BULK_RESULTS_RATE_LIMIT_MAX`). ## Response ```json { "results": [ { "id": "snp_abc:overall", "snapshotId": "snp_abc", "experimentId": "exp_abc123", "phase": "0", "type": "standard", "triggeredBy": "schedule", "dateCreated": "2026-01-14T04:11:02.000Z", "dateStart": "2026-01-01T00:00:00.000Z", "dateEnd": "2026-01-14T00:00:00.000Z", "dimension": { "type": "none", "precomputed": false }, "settings": { "datasourceId": "ds_abc", "assignmentQueryId": "user_id", "experimentId": "homepage-cta", "segmentId": "", "queryFilter": "", "inProgressConversions": "include", "attributionModel": "firstExposure", "statsEngine": "bayesian", "regressionAdjustmentEnabled": true, "goals": [ { "metricId": "met_signup", "effectiveSettings": { "windowType": "conversion", "windowValue": 72, "windowUnit": "hours", "delayValue": 0, "delayUnit": "hours", "properPrior": false, "properPriorMean": 0, "properPriorStdDev": 0.1, "regressionAdjustmentEnabled": true, "regressionAdjustmentDays": 14 } } ], "secondaryMetrics": [], "guardrails": [] }, "results": [ { "dimensionValue": "", "totalUsers": 41230, "checks": { "srm": 0.62 }, "metrics": [ { "metricId": "met_signup", "metricName": "Signups", "variations": [ { "variationIndex": 0, "variationKey": "0", "variationId": "var_control", "variationName": "Control", "users": 20604, "analyses": [ { "engine": "bayesian", "differenceType": "relative", "numerator": 1834, "denominator": 20604, "mean": 0.089, "stddev": 0.284, "effect": null, "ciLow": null, "ciHigh": null, "chanceToBeatControl": null } ] } ] } ] } ] } ], "limit": 10, "offset": 0, "count": 1, "total": 37, "hasMore": true, "nextOffset": 10 } ``` ### Item fields | Field | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `id` | Unique per item: `{snapshotId}:overall`, or `{snapshotId}:dimension:{urlEncodedDimensionId}`. | | `snapshotId` | Shared by all items expanded from the same snapshot. | | `experimentId` | Internal experiment id. Note `settings.experimentId` is the experiment tracking key instead. | | `phase` | Zero-based phase index, as a string. | | `type` | `standard`, `exploratory`, or `report`. `reportId` is present when `report`. | | `triggeredBy` | What ran the snapshot, when recorded. Either `manual` (manually triggered by a user in the UI or API), `schedule` (triggered by the scheduled cron job), `manual-dashboard` (triggered by a user in an Experiment Dashboard), or `update-dashboards` (triggered by the update job that refreshes Experiment Dashboards). | | `dateCreated` | ISO date-time the snapshot was generated. Snapshots are returned newest first. | | `dateStart` / `dateEnd` | Analysis window frozen at snapshot time. | | `dimension` | `{ type, id?, precomputed }`. `type` is `none`, `experiment`, `user`, or a pre-exposure kind such as `date`. | | `settings` | Snapshot-authoritative analysis settings, including per-role metric lists (`goals`, `secondaryMetrics`, `guardrails`, `activationMetric`). | | `results` | One entry per dimension slice: `dimensionValue`, `totalUsers`, `checks.srm`, and `metrics`. | Each metric role entry is `{ metricId, effectiveSettings? }`, where `effectiveSettings` is the snapshot's computed metric settings (conversion window, delay, prior, regression adjustment, `targetMDE`). It is omitted for snapshots generated before those were captured. ### Variation analyses Each variation carries `variationIndex` and `variationKey` — the authoritative join keys, taken from the snapshot's stored variation array — plus its `users` count and an `analyses` array holding one entry per stored difference type: `engine`, `differenceType` (`relative`, `absolute`, `scaled`), `numerator`, `denominator`, `mean`, `stddev`, `effect`, `ciLow`, `ciHigh`, `pValue`, and `chanceToBeatControl`. Every statistic is `null` when it is missing or non-finite, and `effect` is expressed according to `differenceType`. ## Semantics **Snapshot-authoritative.** `settings`, the metric lists, `effectiveSettings`, the analysis window, and each variation's `variationKey`/`variationIndex` come from the stored snapshot and the analysis that produced the numbers. `variationId`, `variationName`, and `metricName` are best-effort current display metadata resolved by id, so they may reflect later renames or be absent entirely. Legacy snapshots fall back to current experiment values only where the stored field is absent. **One item per dimension.** A snapshot expands into one item per dimension, all sharing `snapshotId`. Only the snapshot's default analysis and its stored difference-type variants are returned; missing variants are not computed on the fly, and ad-hoc analyses with other settings (a different baseline or stats engine) are excluded. **Pagination is over snapshots.** `total` counts matching snapshots and `count` counts the snapshots on this page, so a page of `count: 1` can still return many `results` items. `hasMore` and `nextOffset` advance over snapshots too. Offsets are only stable once the window has closed: a snapshot generated inside the window while you page through it sorts ahead of older ones and shifts later pages. --- ## Experiment Configuration import ExternalLink from '@site/src/components/ExternalLink'; # Experiment Configuration The Experiments section in GrowthBook is where you create experiments, configure their analysis, and review results. Before [analyzing results](/app/experiment-results), you need to run the experiment. GrowthBook supports several ways to do this: - [Feature Flags](/features/feature-flag-experiments) (most common) - Running an inline experiment directly with our [SDKs](/lib) - [URL Redirects](/app/url-redirects) - Our [Visual Editor](/app/visual) - Your own custom variation assignment / bucketing system ## Creating an Experiment GrowthBook uses a slim creation flow: you start by creating a minimal experiment, then configure everything else afterward on the experiment's **Overview** page. To create a new experiment, go to the [**Experiments** ](https://app.growthbook.io/experiments) page and click **Add → Create New Experiment**. The creation modal only asks for the essentials: - **Name** - **Project** (if you use projects) - **Template** (optional, but recommended — see below) - **Holdout** (optional) - **Hypothesis** - **Assignment Attribute** — the attribute hashed together with the tracking key to bucket users into variations - **Custom Fields** — optional [custom fields](/using/growthbook-best-practices#custom-fields) added by your organization to experiments GrowthBook creates the experiment as a **draft** with two default variations and, when possible, auto-selects your data source and experiment assignment table. You are then taken to the experiment's Overview tab, where you finish setup (see [Configuring an Experiment](#configuring-an-experiment)). ### Start From a Template Templates are the fastest way to create a consistent experiment. Selecting a **Template** in the creation modal pre-fills settings like the data source, goal/secondary/guardrail metrics, targeting, and traffic, so there is less to configure afterward. You can also require that all new experiments use a template under **Settings → General → Experiment Settings**. See the [Experiment Templates](/running-experiments/experiment-templates) guide for details. ### Finish Setup With the Pre-Launch Checklist After creation, the draft experiment's Overview tab includes a pre-launch checklist that guides you through the remaining steps, such as: - Selecting a data source and experiment assignment table - Adding at least one goal metric - Adding a linked [Feature Flag](/features/feature-flag-experiments), [Visual Editor](/app/visual) change, or [URL Redirect](/app/url-redirects) The pre-launch checklist can be configured by Enterprise customers, [read more here](/app/pre-launch-checklist). Analyzing an experiment you ran elsewhere? See [Importing existing experiments](/app/importing-experiments). ## Configuring an Experiment Most configuration happens after creation, on the experiment's Overview tab. There are several different ways to configure your experiment analysis. ### Experiment Metadata On the experiment page in the "Overview" tab near the top of the page, you can see the experiment name, tags description, hypothesis, and variation metadata. You can edit these fields as you see fit to help describe and categorize your experiment. ### Experiment Targeting and Traffic You can also configure targeting and traffic to your experiment's linked feature flags or visual editor changes. These settings do not have any effect on an experiment that is performing analysis only (with the exception of "experiment key"). On the "Overview" tab, you will see a section called "Targeting and Traffic" which allows you to modify these settings, such as: **Experiment Key** (tracking key) - This is the key that will be used when filtering your experiment assignment source to query experiment exposure data. **Hashing attribute** - This is the attribute that will be used to hash the user id to determine which variation they will be bucketed into. **Fallback attribute** (Sticky Bucketing enabled only) - The Fallback attribute be used when the hash attribute is missing or empty. For example falling back to an anonymous cookie identifier instead of a logged-in user id. Which ever attribute is first used to bucket the user into a variation will "stick". For example, if the user is logged out when they first view an experiment, it would use the fallback device id. If the user later logs in, it will continue using the bucket from their device id, even though they now have a logged-in id as well. **Targeting** - Create matching conditions using attributes and saved groups or target by namespaces. **Traffic** - Choose the percentage of traffic (coverage) and set the relative weights of each variation. ### Analysis Settings Here is where much of your experiment analysis is configured. On the experiment's Overview tab, this section is accessed via the **Edit Settings** button — the label shown on the button that opens the Analysis Settings form. Many of the fields here have some text explaining how they affect your analysis. Here are a few of them in more detail: :::note Permissions required The **Edit Settings** button is only visible to users with permission to edit the experiment's analysis settings (for example, the `Admin`, `Experimenter`, or `Project Admin` roles, or a custom role with equivalent permissions). If you don't have one of these roles, the button will not appear, and you'll need to ask an organization or project admin to make changes on your behalf. See [User Permissions](/account/user-permissions) for details on roles. ::: **Experiment Key** **Activation Metric** - A binomial metric that will filter the users in your analysis to only those who have converted on this binomial metric. This should only be used if activation is expected to be independent of experiment assignment. If an experiment affects the activation metric directly, then there is the potential for bias in your analysis. **Metric Conversion Windows** - Some of your metrics may have "conversion windows" defined for them. For those metrics, we build a window for each user based on when they were first exposed to the experiment. If a user's first exposure to an experiment was recent (for a running experiment) or or near the end of a stopped experiment, they may not have had the full window to convert before the analysis window closes. You may exclude these users with "In-Progress Conversions" if you want your experiment averages to only include those who have had the full window to convert. **Conversion Override** Previously called "Attribution Model" in the UI, this setting lets you use an experiment level override to disable all conversion windows and instead use the exposure period for a user (from their first exposure until the end of the experiment) as the metric window. You can read more about Metric Windows on the [Metric documentation page](metrics). - **Respect Conversion Windows** - This setting ensures that all conversion windows on your metrics are respected. - **Ignore Conversion Windows** - This setting overrides all metrics to be as if they had no conversion windows. Lookback windows will not be overriden by this setting. For those familiar with he "Experiment Duration" attribution model, choosing "Ignore Conversion Windows" will work the same way, and you should know that you can now specify the metric window behavior on a metric-by-metric basis (see the [Metric documentation page](metrics)). ### Experiment Metrics You can add metrics as goal metrics, guardrail metrics, or both. You also can add "Metric Overrides" which provide experiment-specific controls for metrics, allowing you to override the metric's defaults for, for example, metric windows and risk thresholds. ### Experiment Phases Experiment Phases are used primarily to separate stages of an experiment that are to be analyzed separately (e.g. you needed to restart an experiment and change variation weights). For analyzing by specific date ranges, use [Custom Reports](/app/experiment-results#creating-a-custom-report) instead. Be aware that if you move the start date of your analysis to later than the actual start date of the experiment or a new phase (or users were not re-randomized across phases), then your analysis could suffer from [carryover bias](/kb/experiments/carryover-bias). It is best to always look at the full history of an experiment if possible. ### Making Changes While Running Read our dedicated guide on [making changes to running experiments](/app/making-experiment-changes). --- ## Experiment Dashboards import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; # Experiment Dashboards Use experiment dashboards to create tailored views of your experiment. Highlight key insights, add context, and share a clear story with your team. ![Experiment Dashboard](/images/experiment-dashboard-editor.png) ## Creating Dashboards To create a new dashboard, go to the experiment you want to create a dashboard for and click the **Dashboards** tab. If there are no dashboards yet, click the **Create Dashboard** button. If there are existing dashboards, the first one will be open automatically when you go to the tab. To create a new dashboard, open the select box in the top left, which lets you switch between dashboards. There will be an option there to **Create New Dashboard**. Give the dashboard a name and select a few options: - **Auto-update dashboard data** - If enabled, the dashboard will automatically refresh its data whenever the experiment results are updated. If disabled, the dashboard will only refresh when you manually click the update button. - **View access** - Control whether your organization members can view your new dashboard. - **Edit access** - Share the ability to edit dashboard contents with your organization. Even when editing is shared, only the owner and organization admins can change access permissions. After creating the dashboard, you will be able to add blocks to it. ## Dashboard Blocks An experiment dashboard consists of a set of blocks. There are several types of blocks you can add: - **Markdown**: Add custom formatted text, images, and links to provide context and insights. - **Experiment Metadata**: Display the description, hypothesis, and variation names/screenshots for the experiment. - **Experiment Traffic**: Show the count of units in each experiment variation. Choose between a simple table or a timeseries chart. - **Metric Results**: Include the main experiment results table, with the ability to customize the metrics and columns that are included. - **Dimension Results**: Show a breakdown of a specific metric by a chosen dimension (e.g., country, device type). You can select which metric and dimension to display. - **Time Series**: Display a timeseries chart for specific metrics within an experiment, allowing you to visualize trends over time. - **SQL Explorer**: Show a visualization from a custom SQL query. This is useful for displaying data that isn't directly available in the standard experiment results. ## Use a Dashboard as the Default Experiment View You can set an experiment dashboard as the default view to present a more curated, stakeholder-friendly summary of your experiment results. This is particularly useful for presenting findings from stopped experiments in a way that's easier for non-technical stakeholders to understand. :::note Setting a dashboard as the default view for an experiment will require experiment edit permission. ::: **To set a dashboard as the default view:** 1. Navigate to the experiment's **Dashboards** tab 2. Select the dashboard you want to use 3. Click the **More** menu (⋯) and select **Edit Dashboard Settings** - **View access** must be set to **Organization members** 4. Click the **More** menu (⋯) and select **Set as Default View** **To revert to the standard view:** Follow the same steps but select **Remove as Default View** from the More menu. When a dashboard is set as the default view, you can still access the standard experiment view by clicking **View Regular Experiment View** above the dashboard. --- ## Understanding Experiment Results import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Experiment Results Once your experiment is up and running, you will be able to track how it is performing in the Experiment Results tab. This can be found on any Experiment page under the Results tab at the top. There, you can update your analysis, configure how you want your analysis to run, and see the impact of your experiment on your metrics. ## Experiment Results Table The heart of the experiment results page is the table of results. What you will see will depend a little bit on whether you are using our Bayesian or our Frequentist engine, as well as whether your experiment has 2 variations or if it has 3+, but in either case the first two data columns will look the same, and each row will represent one comparison between baseline and variation for one metric. ![Results Table](/images/results-table.png) In both engines, the first two data columns are: **Baseline** - the average value of the metric in the baseline variation; either a percentage of users in that variation (for proportion, retention, and funnel metrics) or an average value for mean or ratio metrics. **Variation** the average value of the metric in the comparison variation. For both columns, the raw data in grey underneath shows the numerator and denominator totals. ### Bayesian Engine **Chance to Win** tells you the probability that the variation is better. Anything above 95% (a customizable threshold set at the organization level) is highlighted green indicating a very clear winner. Anything below 5% is highlighted red, indicating a very clear loser. Anything in between is grayed out indicating it's inconclusive. If that's the case, there's either no measurable difference or you haven't gathered enough data yet. The graph and the **% Change** column show you how much better/worse the variation is compared to the baseline. It is a probability density graph and the thicker the area, the more likely the true percent change will be there. As you collect more data, the tails of the graphs will shorten, indicating more certainty around the estimates. ### Frequentist Engine If you select the "Frequentist" engine, when you navigate to the results tab to view and update the results, you will see the following results table: ![Results Table (Frequentist)](/images/results-table-frequentist.png) The **P-value** column is the probability that the experiment effect for a variation would have been observed if the true effect was zero. When the p-value is less than 0.05 (a customizable threshold set at the organization level) and the experiment effect is in the preferred direction, we highlight the cell green, indicating it is a clear winner. When the p-value is less than 0.05 and the experiment effect is _opposite_ the preferred direction, we highlight the cell red, indicating the variant is a clear loser on this metric. The graph now represents a 95% confidence interval (or 100\*(1 - $\alpha$)% confidence interval if you have a custom significance threshold other than 0.05). The _% Change_ column is unaffected, although we now also represent the width of the confidence interval in grey. ## Metric Drilldown Clicking any metric or slice row in the results table opens the **Metric Drilldown** modal, which provides a focused view of that metric, its slices, and more debugging information. The modal includes three tabs: - **Overview** - Metric details and [time series](/app/experiment-time-series) for the metric, a link to the metric page, a results table for that metric with analysis controls (baseline, variation filter, difference type), and a metric card showing key information. - **Slices** - All slices for that metric displayed in a searchable, sortable table. Clicking a slice row in the main results table opens the modal with the Slices tab focused and that slice pre-filtered for quick exploration. This is also where time series for metric slices are displayed. - **Debug** - Debug information for power users to show the impact of: CUPED, Post-Stratification, Capping, and informative priors (Bayesian engine only). For [Funnel Metrics](/app/metrics/funnel), a **Funnel** tab replaces the Slices tab. See [Funnel Metric Results](#funnel-metric-results). ![Metric Drilldown modal - Overview tab](/images/experiment-results/metric-drilldown-overview.png) ![Metric Drilldown modal - Debug tab](/images/experiment-results/metric-drilldown-debug.png) For more information on how metric slices are defined, see [Metric Slices](/app/metrics#metric-slices), an Enterprise-only feature. ## Funnel Metric Results [Funnel Metrics](/app/metrics/funnel) measure the share of exposed units who complete an ordered series of steps. In the results table they appear as a single row marked with a funnel icon that expands into one row per step. ### Step rows Click the caret next to a funnel metric to **Expand funnel steps**. Each step gets its own row, labeled with the step name and `Step N` below it, plus `(optional)` for steps you marked optional. Every step row is an ordinary proportion, so it reports a baseline rate, a variation rate, a % change, and significance in the same columns as your other metrics. The parent row measures the share of units who reach the last step, which is the end-to-end completion rate. The denominator is the same for the parent row and for every step row: all exposed units in that variation. Step 3's rate is the share of exposed units who reached step 3, not the share of step 2 completers who continued. For step-to-step conversion, use the funnel chart below. Step rows always stay in funnel order, first step to last, no matter which column you sort by. Sorting reorders the parent metrics and keeps each funnel's steps grouped beneath it. Funnel rows expand the same way in the metric and time series blocks on [Experiment Dashboards](/app/experiment-dashboards). ### Funnel chart Clicking a funnel metric opens the [Metric Drilldown](#metric-drilldown) with a **Funnel** tab in place of the Slices tab. It charts every variation's funnel side by side, one color per variation, with a faded bar stacked above each step showing the drop-off from the previous step. The chart is normalized so that each variation's first step is 100%, which lets you compare shapes across variations of different sizes. Hovering a step shows each variation's unit count, its share of the first step, and its share of the previous step. :::note Average time to convert between steps is not yet available for funnel metrics in experiments. The [Funnel explorer](/app/product-analytics/explorer#funnels) reports it for ad-hoc funnels. ::: ### Dimensions and date results Funnel metrics support dimensions. When you break results down by [date](#date), each step gets its own chart under the funnel's heading, titled `Step N: `, with `(optional)` appended for optional steps. ### Downloading results CSV exports include one row per step alongside the funnel itself, with step rows named `Funnel name: Step name`. The [bulk experiment results API](/app/experiment-bulk-results-api) reports the same, using per-step metric ids of the form `?step=`. ### Funnel limitations - **CUPED and regression adjustment** are always off for funnel metrics and their steps. Support is coming soon! - **Metric slices** are not supported. The Slices tab is replaced by the Funnel tab. - **Activation metrics** cannot be funnel metrics. - **Incremental Refresh** does not support funnel metrics while the incremental pipeline is in beta. Support is coming soon! ## Guardrails Guardrail metrics are treated much like regular metrics, but they are placed in a separate part of the results view and are not part of any p-value corrections in the frequentist engine (in other words, even with p-value corrections applied, these results will be more sensitive to negative or positive trends). They also can be used differently from regular metrics in the [Experiment Decision Framework](/app/experiment-decisions). ## Results Table Settings There are several settings at the top of the results table that allow you to control your results. ### Variations This option allows you to filter which variations are shown in the results table, in the case that you have 3+ variations in your experiment. ### Baseline Variation This option allows you to change which variation is the baseline variation. This is particularly useful in the case when you have one control and two treatment variations. In this case, our result defaults to showing you the statistics comparing each treatment variation versus the baseline variation, but you may want to additionally analyze how the treatment variations compare to one another. In that case, you can switch the baseline to be one of the treatment variations to directly compare treatment 1 to treatment 2. ### Difference Types A "difference type" is the way we measure the difference in variations. There are three difference types that you can select from" - `Relative` - The default, this is the relative change or "uplift" of your variation when compared to the baseline. Specifically, the `Relative` change is $100\% * \frac{\mu_t - \mu_c}{\mu_c}$ where $mu_t$ and $\mu_c$ are the averages in the treatment and baseline variations respectively. Effects here tell you that the average user value in the variation was X% greater than the average user value in treatment. For example, if your metric is Revenue and your baseline average is 10.00 and your variation average is 10.31, then your `Relative` change is 3.1\%. - `Absolute` - This is simply the difference in average values across your variations --- $\mu_t - \mu_c$. This can help you understand the raw difference in average values, e.g. the treatment leads to an increase in revenue of \$0.31 per user in the above example. - `Scaled Impact` - This helps you understand the daily total (as opposed to average) effect that your experiment would have had if 100% of users that would have been exposed to your treatment variation had gotten that treatment variation. It is computed as $(\mu_t - \mu_c) * (N_t / p_t) * (1 / d)$, where $N_t$ is the number of users that are in your treatment variation, $p_t$ is the percent of all traffic that is in that variation, and $d$ is the number of days in the current phase used for the results. So if your experiment ran on 10% of traffic for 20 days, with 5% going to the treatment variation, and there were 5,000 users in your treatment variation, the scaled impact would be $0.31 * 5,000 / 0.05 / 20 = 1,550$ dollars per day. This implies that this experiment would have lead to a $1,550$ increase in total revenue per day if every user that could have been exposed the variation had been exposed to the variation. These difference types can have slightly different statistics because for Relative effects we need to account for the uncertainty in estimating $\mu_c$ (which forces us to use a delta method derived variance to properly handle). For more details, see the [Statistical Details](/statistics/details). Furthermore, with CUPED enabled, you may find that the changes are not exactly the same as the difference in raw variation averages, due to CUPED adjusting those averages under the hood to reduce variance. ### Dimensions Dimensions allow you to slice and dice your data, but require additional queries to compute. #### User or Experiment If you have defined dimensions for your data source, you can use the **Unit Dimension** dropdown to drill down into your results. For SQL integrations (e.g. non-MixPanel) GrowthBook enforces one dimension per user to prevent statistical bias and to simplify analyses. For more on how GrowthBook picks a dimension when more than one are present for a user, see the [Dimensions documentation](/app/dimensions). This is very useful for debugging (e.g. if Safari is down, but the other browsers are fine, you may have an implementation bug) or for better understanding your experiment effects. Be careful. The more metrics and dimensions you look at, the more likely you are to see a false positive. If you find something that looks surprising, it's often worth a dedicated follow-up experiment to verify that it's real. #### Date The date dimension shows a time series of the count of users _first_ exposed to an experiment, as well as effects when comparing users _first_ bucketed on each day. Take the following results, for example. ![Experiment Date Results](/images/experiment-date-results.png) In the first graph, we see the number of users who were first exposed to the experiment on that day. In the second graph, we see the uplift for the two variations relative to the control _for all users first bucketed on that day_. That means that the values on October 7 show that users first exposed to the experiment on October 7 had X% uplift relative to control, when pooling all of the data in their relevant metric window. It does not mean show the difference in conversions across variations on October 7 for all previously bucketed users. ## Experiment Health By default, GrowthBook runs several health checks on your experiment to help you diagnose if there are any issues with the configuration. Health checks appear in two places in the app: 1. On the results tab -- we insert Experiment Balance and Multiple Exposures warnings for all experiments that might be untrustworthy right into your results tab 2. On the health tab -- if you have health traffic queries enabled, we can run additional queries to further dig into any experiment health issues ### Experiment Balance (SRM) Every experiment automatically checks for a Sample Ratio Mismatch (SRM) and will warn you if found. This happens when you expect a certain traffic split (e.g. 50/50) but you see something significantly different (e.g. 46/54). We only show this warning if the p-value is less than `0.001` (customizable in your Organization Settings), which means it's extremely unlikely to occur by chance. ![SRM Warning](/images/srm.png) Like the warning says, you shouldn't trust the results since they are likely misleading. Instead, find and fix the source of the bug and restart the experiment. You can find more information about potential sources of the problems in our [troubleshooting guide](/kb/experiments/troubleshooting-experiments). Under the hood, we are conducting a standard chi-squared test for Sample Ratio Mismatch, which compares the distribution of observed units to the expected units and computes a p-value for the probability of observing this traffic split if the traffic split were truly unbiased. ### Pre-Exposure Bias Check This alert indicates that there is a large difference between the pre-exposure mean of the baseline variation and the pre-exposure mean of a test variation. This means that the baseline and test groups were substantially different with respect to a goal or guardrail metric before the experiment started. While some degree of imbalance is expected, this alert is raised if the imbalance is statistically significant. This matters because large pre-exposure mean imbalances can bias results, leading to incorrect shipping decisions. If SRM or multiple exposures fails, then these should be investigated first. If the pre-exposure mean imbalance is not caused by SRM or multiple exposures, then potential causes include: 1. outliers in the pre-exposure metrics: if a few extreme values are causing the imbalance, then consider [capping](/app/metrics#capped-value) your metric. 2. incorrect timestamps for aggregated data: if you are using daily [pre-aggregated data](/app/metrics/examples#pre-aggregated-tables), please ensure your daily metric timestamps are at the end of the day, rather than the start. This ensures that metric data from the first day of exposure always comes after exposure timestamps. 3. exposure latency: if there is a lag between when the exposure event happens and when it is recorded in the events table, then post-exposure metrics could be counted as pre-exposure. If your lag is large and cannot be reduced, then consider [metric delays](/app/metrics#metric-delays), which shift the pre-exposure window to exclude the lag period. 4. mismatch between randomization unit and analysis unit causing unreliable inference: if you are randomizing on session_id but analyzing by user_id, then attribution of the metric outcomes to multiple sessions for the same user can imbalance results. We use the frequentist engine with sequential testing disabled to calculate the p-values for the pre-exposure mean imbalance, with a significance threshold of 0.001. We calculate the p-values for the pre-exposure mean imbalance using a two-sample t-test on the absolute difference in means. We adjust for testing multiple metrics using a Bonferroni correction including all goal and guardrail metric-variation pairs as standalone tests. We exclude secondary metrics from causing the overall health status to fail as well as from contributing to the Bonferroni-corrected threshold. This means if you have 3 arms (2 variation test arms), 2 goal metrics, 35 secondary metrics, and 2 guardrail metrics, we would correct for a total of (2 goal _ 2 arms + 2 guardrail _ 2 arms) = 8 total tests. Note: we exclude metric slices and quantile metrics from the pre-exposure bias check. ### Multiple Exposures This alert indicates that there a substantial number of users (or other unit) in your experiment that have been exposed to multiple variations. At its core, this means that your Experiment Assignment Query is returning data that has rows that look like the following: | user_id | timestamp | experiment_id | variation_id | | ------- | ------------------- | -------------- | ------------ | | 123 | 2022-08-23-10:53:04 | my-button-test | 0 | | 123 | 2022-08-23 10:53:06 | my-button-test | 1 | This indicates that for some reason your identifier type (in this case `user_id`) is being tracked with multiple values of `variation_id` and it is impossible to tell which variation to assign to that user. This can happen if: - Your SDK is misaligned with GrowthBook's identifier types for some reason. For example, it's possible that the identifier type you're using as the hash attribute in the SDK is different from the one you're firing in the `trackingCallback` to your warehouse. Ensure that the id you're using to hash users is the same one that you see in your data warehouse for a given user. - If you're using something other than GrowthBook for experiment assignment, there are many possible reasons for this issue: - A bug with the third party solution's hashing algorithm - A mismatch between the identifier type that you have set up in GrowthBook and the ID that is being used to assign variations in that third-party solution - A mistaken in your Experiment Assignment Query that is returning the wrong `variation_id` or identifier type for a given experiment. ### Health Tab :::tip To get access to the health tab, you need to enable us to run one additional query per experiment analysis. You can do this on a health tab for any experiment if you have the requisite permissions to run queries. ::: The health tab provides you with more insights on the traffic to your experiment over time and across dimensions. ![Experiment Health Tab](/images/health-tab.png) **Experiment Traffic** - A plot of experiment units by the first date they were exposed to the experiment. You can look at daily traffic or cumulative traffic. If you have your experiment dimensions configured with pre-defined slices, we will also return traffic splits by dimension in this tab. For example, if you have a `browser` dimension with pre-defined slices, we will show you the traffic splits by browser. **Experiment Balance Check** - A table with information on the actual number of experiment units, the expected number, and the differences between the percent traffic allocated to each bucket. We also provide you with checks by any pre-defined dimension slices you have configured for your experiment dimensions. #### Adding Dimensions to Health Tab Your health tab shows dimension breakdowns only for Experiment Dimensions which have pre-defined slices. When setting up the health tab, you will be prompted to configure your Experiment Dimensions to have pre-defined slices that can be used in the health tab. This is optional, but we require pre-defined slices for your experiment dimensions to compute dimension traffic and health checks so that we can run only one additional query per analysis and get reliable results. If you want to refresh your dimension slices or change your dimension definitions, you should do so via the Data Source page for the related data source. You can read more about that [here](/app/dimensions#experiment-dimensions). ## Sharing Experiment Insights Share insights for A/B tests or bandits in 2 ways: Share the full experiment or create a custom report. ![Overflow menu with sharing options highlighted](/images/reports-share-menu.webp) ### Sharing a Full Experiment Share a live view of your experiment that automatically updates with the latest data. Viewers can explore the experiment details, variations, and metrics without making changes. ![Share experiment modal](/images/reports-share-modal.webp) To share an experiment: - Go to the experiment page - Click the overflow menu (⋮) - Select **Share Experiment** - Choose **View access** - **Only organization members:** Available to anyone in your organization - **Anyone with the link:** Available via public link :::tip Changing access from **Anyone with the link** to **Only organization members** will invalidate existing public links. ::: ### Creating a Custom Report Create point-in-time snapshots to capture specific analyses or time periods. Custom reports are ideal for documenting important findings or sharing focused analyses. To create a shareable custom report: - Go to the experiment page - Click the overflow menu (⋮) - Click **Create shareable report** - Click **Edit Analysis** to customize content - To edit report name and description, click the overflow menu (⋮) → **Edit Name & Description** - Click **Share** - Choose **View access** - **Only me**: Private report visible only to you. - **Only organization members**: Visible to your organization. - **Edit access**: Allow specific organization members to modify the report Reports can be manually updated to reflect current data or any specific date range. ### Managing Custom Reports Access your reports from the **Custom Reports** section in the experiment's **Results** tab. ![Manage custom reports via the Results tab](/images/reports-list.webp) From here you can: - View existing reports - Edit report contents - Manage sharing permissions - Delete reports --- ## Experiment Time Series import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Experiment Time Series The time series chart shows the cumulative results of your experiment. It's accessible from the [Metric Drilldown](/app/experiment-results#metric-drilldown) available from the experiment results view. Click on the row for the metric whose time series you want to see. Each data point represents cumulative results for that day. When there are multiple updates within a day, GrowthBook consolidates them into a single daily point to keep the chart easy to read. ![Time Series inside of a Metric Drilldown](/images/experiment-results/metric-drilldown-time-series.png) Selecting a difference type (Relative, Absolute, Scaled Impact) will change the y-axis of the time series to the relevant effect. Selecting a different baseline variation will hide the time series as they are only computed for the default baseline variation. ## FAQ ### Why is some part of the time series dotted? A dotted line in the time series indicates a period when metric settings or analysis settings were different from the current settings or results. Because older data may not be directly comparable, the dotted line shows historical results where comparability is less certain. (A solid line line indicates results that are immediately comparable.) ![Dotted part of the time series](/images/experiment-time-series-dotted.png) ### Why aren't the intervals between data points not equal? The time series reflects experiment refreshes, not fixed time intervals. If updates occur irregularly, the spacing between points will vary. ### Why does the time series only go back so far? The time series will only go back as far as the oldest experiment update (and as far back as late March 2025, when the time series feature was launched). If you have not manually refreshed the experiment results, then the time series will only go back to the last time the experiment was refreshed. Furthermore, if you add a metric mid-experiment, the time series will only show the history of the metric as long as the metric has been in the experiment. --- ## Managed Warehouse # Managed Warehouse GrowthBook Cloud offers a fully managed data warehouse and event tracking pipeline. Provision it in one click, send events from your app, and GrowthBook handles the rest. No infrastructure setup required. :::info Not sure if Managed Warehouse is right for you? See [Choose Your Data Path](/app/choose-data-path) for a side-by-side comparison with connecting your own warehouse. ::: ## What you get When you provision the Managed Warehouse, GrowthBook automatically sets up your data source with everything pre-configured: ### Tables We create 3 tables in a ClickHouse database managed by GrowthBook: | Table | What it stores | Indexed by | | -------------------- | ------------------------------------------------------- | ---------------------------- | | **events** | All custom events (page views, purchases, clicks, etc.) | `event_name`, `timestamp` | | **experiment_views** | Experiment exposure events | `experiment_id`, `timestamp` | | **feature_usage** | Feature flag evaluation events | `feature`, `timestamp` | ### Fact Table An **Events** fact table is pre-built on top of the `events` table. This is the base you use to define metrics. It includes 16+ standard columns (timestamp, event name, device ID, geo and browser data, etc.) plus an `attributes` JSON column with everything your events send, and is ready to use immediately. ### Starter metrics Three metrics are auto-created to get you started: | Metric | Type | What it measures | | ----------------------- | ----- | ------------------------------ | | **Page Views per User** | Mean | Average number of page views | | **Sessions per User** | Mean | Average number of sessions | | **Pages per Session** | Ratio | Page views divided by sessions | You can edit these or create your own. Read more about [metrics and fact tables](/app/metrics). ### Default identifiers and dimensions Identifiers are the columns used to split traffic in experiments, and dimensions let you slice and dice results. These are set up out of the box: - **Identifiers:** `user_id`, `device_id` - **Dimensions:** `geo_country`, `ua_browser`, `ua_os`, `ua_device_type`, `utm_source`, `utm_medium`, `utm_campaign` You can add more identifiers through your organization's attribute settings. See [Identifiers and custom attributes](#identifiers-and-custom-attributes) for details. ## Getting started ### 1. Provision the Managed Warehouse Go to **Metrics and Data** → **Data Sources** and click **Create** on the Managed Warehouse option. GrowthBook provisions your ClickHouse database, tables, fact table, and starter metrics in seconds. ### 2. Install the SDK with tracking Add the GrowthBook SDK to your application with the tracking plugin enabled. Here are the two most common setups: :::info Using eu-west-1? Events default to the `us-east-1` ingestor. If you selected **eu-west-1** as your Data Region, add the `ingestorHost` (or `data-event-ingestor-host` for the script tag) override — see [Sending events to the right region](#sending-events-to-the-right-region) below — otherwise your events won't reach your warehouse. ::: **HTML Script Tag**: add one line to your page: ```html ``` **JavaScript / React**: install the SDK and enable the tracking plugin: ```js import { GrowthBook } from "@growthbook/growthbook"; import { autoAttributesPlugin, growthbookTrackingPlugin } from "@growthbook/growthbook/plugins"; const gb = new GrowthBook({ clientKey: "YOUR_CLIENT_KEY", plugins: [ autoAttributesPlugin(), growthbookTrackingPlugin() ] }); ``` The tracking plugin automatically sends feature usage and experiment view events. You can also log custom events: ```js gb.logEvent("Purchase", { amount: 49.99 }); ``` See [all supported SDKs](#with-sdks) for more options including Node.js, Python, Java, PHP, Go, and Flutter. ### 3. Verify events are flowing Open the **SQL Explorer** in GrowthBook and run a quick query to confirm events are arriving: ```sql SELECT event_name, COUNT(*) AS count FROM events WHERE timestamp >= now() - INTERVAL 1 HOUR GROUP BY event_name ORDER BY count DESC ``` You should see your events appear within seconds of sending them. ### 4. Create your first metric Your auto-created metrics (Page Views per User, Sessions per User, Pages per Session) are ready to use immediately. To create a custom metric: 1. Go to **Metrics and Data** → **Fact Tables**. 2. Open the **Events** fact table. 3. Click **Add Metric** and configure it (for example, a Proportion metric filtered to `event_name = 'Purchase'` to track purchase conversion rate). Read more about [metric types and configuration](/app/metrics). ### 5. Run your first experiment With events flowing and metrics defined, you're ready to experiment: 1. Create a feature flag in **Features**. 2. Add an **Experiment** rule to it. 3. Start the experiment and watch results flow in on the **Results** tab. For a detailed walkthrough, see our [guide to running Feature Flag Experiments](/features/feature-flag-experiments). --- ## Benefits - **Fully managed**: No infrastructure management, scaling, or maintenance. - **Seamless integration**: One-click setup + built-in tracking in our SDKs. - **Instant data**: Events are enriched and available within seconds. - **Raw SQL access**: Use the SQL Explorer to run custom queries against your data. ## How it works We use ClickHouse, a database optimized for real-time analytics, to store your event data. The process is: 1. You send analytics events to our scalable ingestion API. 2. We enrich and store them in ClickHouse within seconds. 3. You can query the data with SQL, define metrics, and analyze experiment results with our stats engine. ## Sending events There are 2 ways to send events to GrowthBook Cloud's Managed Warehouse: 1. With our [SDKs](#with-sdks) (limited language support). 2. With our [Ingestion API](#ingestion-api). ### With SDKs The following SDKs have a built-in plugin to automatically send events. - [HTML Script Tag](#html-script-tag) - [Client-Side JavaScript / React](#client-side-javascript--react) - [Node.js](#nodejs) - Python - Java - PHP - Golang - Flutter - Swift (Coming Soon) - Kotlin (Coming Soon) For everything else, use the [Ingestion API](#ingestion-api). When the plugin is added, these SDKs will automatically send feature usage and experiment view events to GrowthBook. They also expose a helper method to log additional custom events with optional properties. All of the attributes in the SDK are sent along with events as context. Make sure you do not include any sensitive information in your attributes (or if you do, anonymize it first). #### HTML Script Tag Add `data-tracking="growthbook"` to your script tag to enable. ```html ``` To track additional events, use the `window.gbEvents` global variable. You can push events to this array, and they will be tracked. ```html // Ensure the global variable exists window.gbEvents = window.gbEvents || []; // Simple (no properties) window.gbEvents.push("Page View"); function handleSignUpClick() { // With custom properties window.gbEvents.push({ eventName: "Button Click", properties: { button: "Sign Up" } }); } Sign Up ``` #### Client-Side JavaScript / React Use the `growthbookTrackingPlugin` to enable tracking. We recommend also using the `autoAttributesPlugin` to include many common attributes in your events (browser, session_id, etc.). ```js import { GrowthBook } from "@growthbook/growthbook"; import { autoAttributesPlugin, growthbookTrackingPlugin } from "@growthbook/growthbook/plugins"; const gb = new GrowthBook({ clientKey: "YOUR_CLIENT_KEY", plugins: [ autoAttributesPlugin(), growthbookTrackingPlugin() ] }); ``` Use the `logEvent` method to track additional custom events: ```js // Simple (no properties) gb.logEvent("Page View"); // With custom properties gb.logEvent("Button Click", { button: "Sign Up", }); ``` #### Node.js Use the `growthbookTrackingPlugin` to enable tracking: ```js import { GrowthBookClient } from "@growthbook/growthbook"; import { growthbookTrackingPlugin } from "@growthbook/growthbook/plugins"; const gb = new GrowthBookClient({ clientKey: process.env.GROWTHBOOK_CLIENT_KEY, plugins: [growthbookTrackingPlugin()], }); ``` Use the `logEvent` method to track additional custom events: ```js gb.logEvent("Sign Up", { accountPlan: "pro" }, userContext); ``` User-scoped instances also have a `logEvent` method that doesn't require the user context: ```js req.growthbook.logEvent("Sign Up", { accountPlan: "pro" }); ``` #### Sending events to the right region The tracking plugin sends events to `https://us-east-1.gb-ingest.com` by default. If you selected **eu-west-1** as your Data Region when creating your Managed Warehouse, override the ingestor host so events reach the right ClickHouse cluster instead of being dropped: - **HTML Script Tag**: add `data-event-ingestor-host="https://eu-west-1.gb-ingest.com"` to the script tag. - **JavaScript / React / Node.js**: pass `ingestorHost` to `growthbookTrackingPlugin()`: ```js growthbookTrackingPlugin({ ingestorHost: "https://eu-west-1.gb-ingest.com", }) ``` You can find your datasource's configured region on its settings page. ### Ingestion API You can also send events directly to our ingestion API. The host depends on the **Data Region** you chose when creating your Managed Warehouse datasource: - `us-east-1` → `https://us-east-1.gb-ingest.com` - `eu-west-1` → `https://eu-west-1.gb-ingest.com` Sending events to the wrong region's host means they won't reach the ClickHouse cluster your data warehouse actually lives in — always match the host to the region shown on your datasource's settings page. Pass an array of event objects, each with the following properties: - **event_name**: The name of the event (e.g., "Purchase", "Button Click") - **properties**: Optional key-value pairs with properties of the event itself - **attributes**: Optional key-value pairs with attributes of the user or context at the time of the event - **timestamp**: Optional ISO timestamp of when the event occurred. Only used when batching events with a `sentAt` field (see [Event timestamps](#event-timestamps) below). ```bash curl -X POST "https://YOUR_WAREHOUSE_REGION.gb-ingest.com/track?client_key=YOUR_CLIENT_KEY" \ -H "Content-Type: application/json" \ -d '[{ "event_name": "Purchase", "properties": { "amount": 100 }, "attributes": { "user_id": "12345" } }]' ``` Make sure you do not include any sensitive information in your events (or if you do, anonymize it first). #### Identifiers To tie events to users for experiment analysis, include identifier keys in `attributes`: - **`user_id`**: a logged-in user ID. Promoted to the top-level `user_id` column at ingest. - **`device_id`**: an anonymous device or browser ID. Promoted to the top-level `device_id` column at ingest. - **`anonymous_id`** or **`id`**: also resolve to the `device_id` identifier, but at analysis time instead of ingest — they stay inside the `attributes` JSON. Prefer the explicit `user_id` and `device_id` keys, and use each key for one kind of ID consistently. For example, if you send a logged-in user ID as `id` today and later start sending a real device ID as `device_id`, the `device_id` identifier will mix the two ID spaces and experiment analysis will silently drop units. See [Identifiers](#identifiers-and-custom-attributes) for how these keys map to identifiers. #### Event timestamps By default, events are timestamped with the time they are received by the ingestion API. If you batch events before sending, you can preserve each event's original timing by adding a per-event `timestamp` along with a top-level `sentAt` field: ```bash curl -X POST "https://YOUR_WAREHOUSE_REGION.gb-ingest.com/track?client_key=YOUR_CLIENT_KEY" \ -H "Content-Type: application/json" \ -d '{ "sentAt": "2025-06-01T12:00:05.000Z", "events": [{ "event_name": "Purchase", "timestamp": "2025-06-01T12:00:00.000Z", "properties": { "amount": 100 }, "attributes": { "user_id": "12345" } }] }' ``` - **sentAt**: ISO timestamp of when the request was sent, from the same clock used for the event timestamps - **timestamp**: ISO timestamp of when the event occurred. Must be earlier than `sentAt`, otherwise it is ignored. We don't trust client timestamps directly. Instead, we subtract the difference between `sentAt` and each event's `timestamp` from our own server time when the request is received. This preserves the relative timing of your events while correcting for client clock skew. #### Experiment view events In order to use GrowthBook's experiment analysis features, you must send an event every time a user views an experiment. It must match the following format: - **event_name**: Must be `"Experiment Viewed"` - **properties**: Must include the following key/value pairs: - `experimentId`: The ID of the experiment being viewed - `variationId`: The ID of the variation that was shown to the user In addition, you must include `attributes` with the user attributes that were used to evaluate the experiment, plus any attributes you want to use as dimensions for slicing and dicing. At minimum this must include the attribute the experiment hashes on (its **Assign Variation by Attribute**), since that's the [identifier](#identifiers) used to join exposures to your metrics. #### Feature usage events To take advantage of GrowthBook's feature usage analytics, you must send an event every time a feature is evaluated with a specific format. - **event_name**: Must be `"Feature Evaluated"` - **properties**: Must include the following key/value pairs: - `feature`: The name of the feature being evaluated - `value`: The feature's value that was returned from the evaluation - `source`: (optional) The source of the feature value (e.g., "defaultValue", "experiment", "force") - `ruleId`: (optional) The ID of the specific rule that was used to evaluate the feature (or `$default` if the default value was used) - `variationId`: (optional) If the value came from an experiment, the ID of the variation that was returned #### Attributes Attributes are key-value pairs that provide context about the user or environment at the time of the event. They can be used to slice and dice your data in analysis. It's recommended to include the same attributes you use in your GrowthBook SDK. Some attributes are automatically enriched by the ingestion API: - `ip`: a geoip lookup is done and the following attributes are added. If an ip attribute is not provided, we use the IP address of the request. - `geo_country` - `geo_city` - `geo_lat` - `geo_lon` - `ua`: the user agent is parsed and the following attributes are added. If a user agent attribute is not provided, we use the user agent of the request. - `ua_browser` (e.g. Safari) - `ua_os` (e.g. macOS) - `ua_device_type` (e.g. mobile) - `url`: the URL is parsed and the following attributes are added: - `url_path` (e.g. /products/123) - `url_host` (e.g. www.example.com) - `url_query` (e.g. ?utm_source=google) - `url_fragment` (e.g. #section1) It's also recommended to include attributes about which SDK is being used. This helps with debugging. - `sdk_language` (e.g. python) - `sdk_version` (e.g. 1.2.3) #### Limits When calling the ingestion API directly, be aware of the following default limits: - Maximum of 100 events per request. - Maximum of 1 request per second. If you need to send more than this, reach out and we can increase your limits. ## Identifiers and custom attributes All attributes sent with your events are stored in a native ClickHouse JSON column called `attributes` on the `events` table (event properties live in a `properties` JSON column). Every attribute is automatically queryable — no setup required. Identifiers — the attributes used to split traffic in experiments — come in two flavors: ### Built-in identifiers Every table has `user_id` and `device_id` columns. The columns always exist, but they only carry values when your events include one of the following attribute keys: | Identifier | Populated from attribute keys (first non-empty wins) | Intended for | | ----------- | ---------------------------------------------------- | --------------------------------- | | `user_id` | `user_id` | Logged-in user IDs | | `device_id` | `device_id`, `anonymous_id`, `id` | Anonymous, device, or session IDs | The SDK tracking plugin applies this mapping automatically from your SDK attributes, and the [Ingestion API](#identifiers) does the same for the `user_id` and `device_id` keys. Events that carried their ID as `anonymous_id` or `id` are resolved into `device_id` when queries run, so experiment analysis works with any of these keys — including for past events. When you assign an experiment by hash attribute, GrowthBook picks the matching identifier automatically: experiments assigned on the `id` or `anonymous_id` attribute use the `device_id` identifier for analysis. ### Custom identifiers Any other scalar attribute marked as an **Identifier** under **SDK Connections** → **Attributes** (for example `account_id`) is automatically exposed as a top-level column on your fact tables and available for experiment assignment. :::note Identifier resolution reads from the `attributes` JSON column, so marking an attribute as an identifier applies retroactively to all past events that included it. ::: ## SQL Explorer You can use the SQL Explorer to run ad-hoc queries against your events. This is useful for exploring your data, debugging issues, or creating custom reports. :::note The SQL Explorer only allows read-only SELECT queries. Write operations (INSERT, UPDATE, etc.) are prevented from being executed by the platform. ::: You will see 3 tables in the SQL Explorer: - **feature_usage**: contains all feature usage events - **experiment_views**: contains all experiment view events - **events**: contains all other events ### SQL best practices #### Use indexes For best performance, take advantage of the indexed columns in each table: - **events** table: - `timestamp`: the time the event occurred - `event_name`: the name of the event (e.g., "Purchase", "Button Click") - **feature_usage** table: - `timestamp`: the time the event occurred - `feature`: the name of the feature being evaluated - **experiment_views** table: - `timestamp`: the time the event occurred - `experimentId`: the ID of the experiment being viewed #### Querying attributes and properties The `attributes` and `properties` columns are native ClickHouse JSON columns, so you can access fields inside them with dot syntax. Cast the value to get a concrete type: ```sql SELECT attributes.account_plan::Nullable(String) AS account_plan, toFloat64OrNull(properties.amount::Nullable(String)) AS amount FROM events WHERE timestamp >= '2025-06-01 00:00:00' AND event_name = 'Purchase' ``` Attributes marked as [Identifiers](#identifiers-and-custom-attributes) are already exposed as top-level columns, so you can reference them directly without the JSON path. #### Identifier columns in custom SQL The physical `user_id` and `device_id` columns only contain values that were promoted at ingest — by the SDK tracking plugin, or by the `user_id`/`device_id` attribute keys on the Ingestion API. Events that carried their ID as `anonymous_id` or `id` keep it inside the `attributes` JSON instead. GrowthBook-generated queries resolve this automatically, but hand-written SQL reads the raw columns. If your custom SQL joins or groups on identifiers — especially in a [custom fact table](/app/metrics) used for experiment analysis — use the same resolution expressions GrowthBook uses: ```sql coalesce( nullIf(user_id, ''), nullIf(attributes.user_id::Nullable(String), '') ) AS user_id, coalesce( nullIf(device_id, ''), nullIf(attributes.device_id::Nullable(String), ''), nullIf(attributes.anonymous_id::Nullable(String), ''), nullIf(attributes.id::Nullable(String), '') ) AS device_id ``` Keep the standard column names (`user_id`, `device_id`): experiment analysis joins your fact table to GrowthBook's exposure queries by identifier, so a fact table that extracts an ID from the JSON under a different name won't match any exposures. #### Large queries There is a limit of 1000 rows in the SQL Explorer. Returning raw events over a large time period will quickly exceed this limit. To work around this, use `GROUP BY` in your queries to aggregate results. One common aggregation is by time intervals. You can use the `toStartOf...` functions in ClickHouse for this. For example, to get daily event counts: ```sql SELECT toStartOfDay(timestamp) AS day, COUNT(*) AS count FROM events WHERE timestamp >= '2025-06-01 00:00:00' GROUP BY day ORDER BY day ASC ``` Whether you have thousands or millions of events, this query will always return a manageable number of rows (one per day). There are also functions like `toStartOfHour`, `toStartOfMonth`, etc. that you can use to group by different time intervals. ## FAQ ### Can I use the Managed Warehouse alongside my own warehouse? Yes. You can add a separate data source for your own warehouse at any time and use both simultaneously. Note that each experiment pulls data from a single data source, so you can't mix metrics from both warehouses within one experiment. ### How do I add custom identifiers like `account_id`? `user_id` and `device_id` are built in. To add another identifier, go to **SDK Connections** → **Attributes** and mark the attribute as an **Identifier**. Include it in the `attributes` of your events and it will be exposed as a top-level column on your fact tables. ### Why are the `user_id`/`device_id` columns empty when I query directly? Those columns are only filled at ingest when events include the `user_id` or `device_id` attribute keys (the SDK tracking plugin does this for you). If your events carried their ID as `anonymous_id` or `id`, it lives in the `attributes` JSON — experiment analysis resolves it into `device_id` automatically, but your own SQL needs to do the same. See [Identifier columns in custom SQL](#identifier-columns-in-custom-sql). ### What happens if I exceed my event limit? You'll see a notification in GrowthBook when you're approaching your limit. On free plans, events stop being tracked for the remainder of the month and reset the following month. On paid plans, overage charges apply. ### Can I export my data? You can query and export results through the SQL Explorer. ### Can I use the Managed Warehouse when self-hosting? The Managed Warehouse is only available on GrowthBook Cloud. Self-hosted instances should [connect their own data warehouse](/warehouses). --- ## Query Optimization import CommercialFeature from '@site/src/components/CommercialFeature'; # Query Optimization GrowthBook is designed to run efficient database queries out-of-the-box, but for large datasets or complex metrics, there are a few settings and techniques you can use to optimize your queries. ## SQL Template Variables GrowthBook runs all SQL through a templating engine (Handlebars) to allow for dynamic SQL generation. This lets you apply advanced query optimization. The most common is adding a date filter. For example: ```sql SELECT timestamp, user_id, amount FROM purchases WHERE timestamp BETWEEN '{{startDate}}' AND '{{endDate}}' ``` Read more about [SQL Templates](/app/sql-templates) and the different variables that are available. ## Fact Tables Fact Tables are a shared SQL definition that can be re-used across multiple metrics. For example, a "Purchases" fact table could be used for both a "Total Revenue" metric and a "Items per Order" metric. If you are still using legacy metrics (where each metric has its own separate SQL definition), you are missing out on important query optimizations and the newest features in GrowthBook. Read more about [Fact Tables](/app/metrics) and how to [convert legacy metrics to Fact Tables](/app/metrics/legacy#migrating-legacy-metrics-to-fact-tables). ## Fact Table Query Optimization Fact Table Query Optimization enables faster, more efficient queries. If multiple metrics from the same Fact Table are added to an experiment, they will be combined into a single SQL query. For data sourcees with usage-based billing, this can result in dramatic cost savings. There are some restrictions that limit when this optimization can be performed: - Ratio metrics where the numerator and denominator are part of different Fact Tables are always excluded from this optimization - If `Exclude In-Progress Conversions` is set for an experiment, optimization is disabled for all metrics - If you are using MySQL and a metric has percentile capping, it will be excluded from optimization In all other cases, this optimization is enabled by default. It can be disabled under **Settings → General → Experiment Settings**. When disabled, a separate SQL query will always be run for every individual metric. ## Data Pipeline Mode Data Pipeline Mode reduces the amount of duplicate data your warehouse needs to scan. When enabled, GrowthBook will write some intermediate tables back to your warehouse with short retention and re-use those across all of the metric queries in an experiment. Currently, this is limited to BigQuery, Snowflake, and Databricks, but we are working on adding support for other data sources soon. Read more about [Data Pipeline Mode](/app/data-pipeline). ## Materialized Views If your metric definitions are complex and involve multiple joins or subqueries, you may want to consider creating a materialized view in your warehouse. Setting up materialized views differs by warehouse, so consult the documentation for your specific warehouse for more information. You can also use a tool like [dbt](https://www.getdbt.com/) to create computed tables that are automatically refreshed on a schedule. ## Pre-Aggregated Tables Pre-aggregated tables include a GROUP BY in the data pipeline to compress raw event-level data down to fewer rows. This is usually done when querying raw data directly is prohibitively expensive. GrowthBook supports pre-aggregated tables as long as they satisfy two requirements: - Must be grouped by both user and date - Pre-aggregated columns can only be basic sums or counts. No averages, percentiles, count distinct, or complex derived formulas that break statistical assumptions. Using pre-aggregated tables for experimentation comes with additional complexities and downsides, so we highly recommend sticking with event-level data whenever possible. Read more about this approach with some [examples and best practices](/app/metrics/examples#pre-aggregated-tables). --- ## SQL Templates import CommercialFeature from '@site/src/components/CommercialFeature'; # SQL Templates SQL in GrowthBook is first run through a templating engine (Handlebars) to allow for dynamic SQL generation. This allows for several advanced use cases, outlined below. ## Date Filters GrowthBook always includes a date filter in the SQL queries it generates, but because the queries are complex with multiple nested subqueries, database engines are not always able to fully take advantage of these filters for optimization. If you have a date-partitioned table, you can use template variables within the SQL you enter in GrowthBook to provide better hints to your database. This applies to metrics, fact tables, and experiment assignment queries. Here's an example of a simple Fact Table definition that uses template variables: ```sql SELECT timestamp, user_id, amount FROM purchases WHERE timestamp BETWEEN '{{startDate}}' AND '{{endDate}}' ``` The following variables are available: - **startDate** - `yyyy-MM-dd HH:mm:ss` of the earliest data that needs to be included - **startDateISO** - `yyyy-MM-dd'T'HH:mm:ss.SSS'Z'` of the startDate in ISO format - **endDate** - `yyyy-MM-dd HH:mm:ss` of the latest data that needs to be included - **endDateISO** - `yyyy-MM-dd'T'HH:mm:ss.SSS'Z'` of the endDate in ISO format There is also a `date` helper you can use with the ISO variables to format the date exactly how you need it. For example `{{date endDateISO "yyyyMMdd"}}` | code | meaning | | ---- | --------- | | yyyy | year | | MM | month | | dd | day | | HH | hour | | mm | minutes | | ss | seconds | | t | timestamp | Here's a full example of this being used: ```sql SELECT user_id as user_id, received_at as timestamp FROM orders WHERE partition_key BETWEEN '{{date startDateISO "yyyyMMdd"}}' AND '{{date endDateISO "yyyyMMdd"}}' ``` :::note The inserted values do not have surrounding quotes, so you must add those yourself (e.g. use `'{{ startDate }}'` instead of `{{ startDate }}`). ::: ## Experiment Id Filter When a Fact Table or Metric query is run as part of an experiment analysis, the `{{ experimentId }}` variable is replaced with the experiment's tracking key. This is useful when a single Fact Table contains rows for many experiments and you need to scope it to the one being analyzed: ```sql SELECT user_id, timestamp, score FROM classifier_scores WHERE sample_type = '{{ experimentId }}' ``` When the same SQL is run outside of an experiment context (for example, when previewing a Fact Table or running a standalone Metric Analysis), `{{ experimentId }}` is replaced with `%`. If you need the query to return rows in both contexts, write the filter as a `LIKE` pattern so `%` can act as a wildcard: ```sql WHERE sample_type LIKE '{{ experimentId }}' ``` If you prefer an exact match in experiment context and no filter outside experiment context, you can conditionally add the clause with `{{#if ...}}`. Since `%` is always non-empty, this example uses `replace` to strip `%` first: ```sql WHERE 1=1 {{#if (replace experimentId "%" "")}} AND sample_type = '{{ experimentId }}' {{/if}} ``` ## Phase Filters Experiments can have multiple phases. By default, we use date ranges to restrict data to the current phase, but this may be imprecise if your experiment configs are cached in your app or data is delayed. For more accurate filtering, we expose a `phase.index` template variable. This is a 0-based index of the current phase, so the first phase is `0`, the second phase is `1`, etc. Here is an example Experiment Assignment Query that is using a phase filter. The phase may not always be available (e.g. when fetching data for an entire experiment), so it must be wrapped in a conditional check: ```sql SELECT user_id, timestamp, experiment_id, variation_id FROM experiment_assignments WHERE timestamp BETWEEN '{{startDate}}' AND '{{endDate}}' {{#if phase.index}} AND phase_index = {{phase.index}} {{/if}} ``` :::note The filter will still be added above even when the phase index is `0`, despite that being a falsy value in most languages. The only time it will not be included is when there is no phase information at all (i.e. it is undefined). ::: ## Custom Field Filters Custom Fields for experiments can be used in SQL templates to add additional filtering logic. For example, if you have a Custom Field called `region`, you can use that in your Fact Table definition to filter data based on the region for a specific experiment. Like phases, Custom Fields are not always defined, so you must wrap them in a conditional block. Here's an example: ```sql SELECT user_id, timestamp, event_type FROM events WHERE 1=1 {{#if customFields.region}} AND region = {{sqlstring customFields.region}} {{/if}} ``` The `1=1` part is a common SQL trick to make appending additional `AND` conditions easier. When the Custom Field is unavailable, the WHERE clause will simple be `WHERE 1=1`, which is always true and does not filter out any data. :::note Because Custom Field values are user-supplied, use the `sqlstring` helper (e.g. `{{sqlstring customFields.region}}`) to safely quote string values. It wraps the value in single quotes and escapes any embedded single quotes, preventing SQL injection and query errors on values like `it's`. Do not manually wrap the value in quotes (e.g. `'{{customFields.region}}'`) as this is unsafe. ::: --- ## Running Experiments on GrowthBook # Running Experiments on GrowthBook GrowthBook has a few different ways to run experiments or AB tests depending on your needs. This guide will walk you through the different ways of running experiments on GrowthBook. ## Server Side and Mobile Experiments Server-side A/B testing, also known as backend or server-side experimentation, is a technique used in software development and web applications to test measure the impact of any code changes or new features. The changes may impact both the user interface and the backend of the application, but the decision about what version to serve a user is decided on the server. This has a number of advantages over client side experiments, specifically, that it allows you to run very complex tests that may involve a lot of different parts of the code, and span multiple parts of your application. It also avoids any issues with flickering that can happen with client side testing. ### Feature Flag Experiments With GrowthBook, the easiest way to do server-side testing is by using feature flags. Each feature flag has conditional logic ([rules](/features/rules)) which controls how a feature should be shown, and if a feature should be shown as part of an experiment. GrowthBook also lets you target any feature or rule based on the [targeting attributes](/features/targeting) you define. With GrowthBook, you can add an experiment rule to a feature that will randomly assign the users based on some hashing attribute into one of your experiment variations. You can read more about feature flag experiment rules [here](/features/rules), or more details on running an [experiment with feature flags](/features/feature-flag-experiments). ### Inline Experiments You can also run server-side experiments by using inline experiments directly with our SDK. This requires no 3rd party requests, as the experiment conditions and settings can be written directly into the code. How you implement in-line experiments depends on the SDK language you're using, you can read more about this from our [SDKs pages](/lib). ## Client Side (Browser) Experiments Client-side A/B testing, also known as frontend or client-side experimentation, is a way to test visual changes to your application. The server returns the same code to all users, and the experiment assignments and variations are handled by the client, typically in [Javascript](/lib/js) or [React](/lib/react). When the client loads your application, our SDK will check to see if the user should be part of any experiments, and if so, assign them and the client code will serve that variations treatment to them. GrowthBook's client side SDKs can handle all of this for you, and you can also serve client-side A/B test via our feature flags, or using our visual editor. Client-side A/B tests are great for testing visual changes to your application, but they do have some drawbacks. One of the most common issues is caused by the delay in loading the specific variation to a user, which may cause a flash or flickering as the experiment loads. This can be reduced by using inline-experiments, or by moving the GrowthBook SDK code higher in the code so it loads at the same time or before the page. Sometimes the hashing attribute may not be available until your tracking software has loaded, and you may need to use a custom tracking id. ### Feature Flags GrowthBook's feature flags can work just as well from the client/browser as it does server side. The feature flag's conditional logic ([rules](/features/rules)) controls how a feature should be shown, and if a feature should be shown as part of an experiment. The same targeting and feature rules apply in exactly the same way to client-side experimentation. ### Inline Experiments Just like with inline server-side experiments, you can run experiments inline on the client. This requires no 3rd party requests, as the experiment conditions and settings can be written directly into the code, and processed by the SDK. How you implement in-line experiments depends on the SDK language you're using, you can read more about this from our [SDKs pages](/lib). ### Visual Editor (WYSIWYG) Experiments GrowthBook has a visual editor for running experiments on the front-end of your website without requiring any code changes. Our visual editor uses the same client-side SDK used for feature flags and A/B testing. ## API / ML Experiments GrowthBook's SDKs works well with anywhere code can run, and as such you can use it in the API or when running machine learning models. And, with our deterministic hashing method for assignment, you can even be sure users get assigned the same variation across your platform without needing to store state from GrowthBook. ## Custom Assignment or 3rd Party Experiments GrowthBook is a modular platform and can be used for experiment analysis if you are using custom assignment code or another experimentation system for randomization of users into variations. As long as the exposure/assignment information is available from within your data warehouse, you can use GrowthBook to analyse the results of your experiments. ## How GrowthBook Assigns Users to Experiments GrowthBook uses a consistent hashing algorithm to assign users to experiments, ensuring that the same user will always receive the same variation as long as the experiment settings (experiment seed and user hashing ID) remain unchanged. This makes it possible to run experiments across multiple pages or applications while maintaining consistent user experiences without storing any state or using additional cookies. In cases where experiment settings do change but consistent assignment is still required, GrowthBook offers a feature called [sticky bucketing](/app/sticky-bucketing), which requires additional configuration. ## Best Practices :::tip Get our official guide to A/B Testing This is a subset of some of the information you can find about running experiments in our [Open Guide to AB Testing](/open-guide-to-ab-testing.v1.0.pdf). ::: ### Running A/A Tests An A/A test is the same as an A/B test, but each variation has no actual difference in the application. This lets you test out that your systems are working correctly, as you should see no significant differences between the variations. We suggest that you first an A/A test to validate your experimentation implementation is correctly splitting traffic, and producing statistically valid results. ### When to Expose Users to Experiments When running an experiment it is best if you can only expose users as close to the actual treatment exposure as possible. This means that if you're testing something like new signup flow, you don't expose all users, including those who never open that window. Including users who did not see the treatment will increase the noise and reduce the ability to detect any differences. If assignment is unavoidably separated from exposure, you can use an activation metric to filter out these un-exposed users from the analysis. ### Avoiding Flickering Flickering with front end or client-side A/B tests is an artifact from all client-side A/B testing tools. This is caused by a delay in loading the specific variation for a user, which may cause a flash or flickering as the experiment loads. This can be reduced by using inline-experiments, or by moving the GrowthBook SDK code higher in the code file, or using server side A/B tests. There are a few other ways to reduce flickering that some platforms utilize. One common "flicker free" technique is to load a white overlay, or just hide parts of the page, as the page loads. The result is that users cannot see any flickering that may be happening beneath the overlay as the variations are loaded. ### Sample Size Understanding experiment power and MDE are important to predict how many samples are required. There are numerous online calculators that can be used to help you predict the sample size. Typical rule of thumb for the lowest number of samples required is that you want at least 200 conversion events per variation. So for example if you have a registration page which has a 10% conversion rate, and you have a 2 way (A and B) experiment that is looking to improve the member registrations, you will want to expose the experiment to at least 4,000 people (2000 per variation). ### Test Duration Due to the natural variability in traffic day to day and hour to hour, experimentation teams will often set a minimum test duration within which a test cannot be called. This helps you avoid optimizing a product for just the users that happen to visit when the test is started. For example, if the weekend traffic of your product is different from the traffic during the week, if you started a test on Friday and ended it on Monday, you may not get a complete picture of the impact your changes have to your weekday traffic. Typical test durations are 1 to 2 weeks, and usually care needs to be taken over holidays. You may also find that a test would need to run for a month or more to get the power required for the experiment. Very long running tests can be hard to justify as you have to keep the variations of the experiment unchanged for duration, and this may limit your team's velocity towards potentially higher impact changes. ### Interaction Effects and Mutual Exclusion When you start having the ability to run a lot of A/B tests, you may start worrying about how tests running in parallel may interact and effect the other results. For example you may want to test a change in the CTA button on your purchase page, and also test changing the price. It can be difficult to figure out if any two tests will meaningfully interact, and many will run the tests in serial in an abundance of caution. However, meaningful interactions are actually quite rare, and keeping a higher rate of experimentation is usually more beneficial. You can run analysis after the experiments to see if there were any interaction effects which would change your conclusions. If you need to run mutually exclusive tests, you can use GrowthBook’s namespace feature. Ensure all experiments within the namespace use the same hash attribute (assignment attribute). ### Experimentation Frequency Having a high frequency of A/B testing is important for running a success experimentation program. The main reasons why experimentation frequency is important are: - **Maximizing chances**: Since success rates are typically low for any given experiment, and large changes are even more rare, by having a high frequency of A/B testing you are maximizing your chance of having impactful experiments. - **Continuous improvement**: A high frequency of A/B testing allows you to continuously improve your website or application. By testing small changes frequently, you can quickly identify and implement changes that improve user experience, engagement, and conversion rates. - **Adaptability**: A high frequency of A/B testing allows you to quickly adapt to changes in user behavior, market trends, or other external factors that may impact your website or application. By testing frequently, you can identify and respond to these changes more quickly, ensuring that your site or app remains relevant and effective. - **Avoiding stagnation**: A high frequency of A/B testing can help you avoid stagnation and complacency. By continually testing and experimenting, you can avoid falling into a rut or becoming overly attached to a specific design or strategy, and instead remain open to new ideas and approaches. --- ## FAQ # FAQ Below are some frequently asked questions about GrowthBook. ## Experiment Assignment ### Do users always get assigned the same experiment variation? GrowthBook SDKs use deterministic hashing to ensure the same user always gets assigned the same variation in an experiment. In a nutshell, GrowthBook hashes together the `hashAttribute` (the user attribute used to assign a variation, ex: user id) and the experiment `seed` which produces a decimal between 0 and 1. Each variation is assigned a range of values (e.g. `0 to 0.5` and `0.5 to 1.0`) and the user is assigned to whichever one their hash falls into. This does mean, if you change the experiment configuration, some users may switch their assigned variation. For example, if someone has the hash `0.49` and you adjust the weights to a 40/60 experiment, the variation ranges become `0 to 0.4` and `0.4 to 1.0`. In this case, the user was previously in the control group, but will now be in the variation. GrowthBook will detect issues like this and will remove users who see both variations from the analysis automatically. However, to keep things simple and safe, we recommend not relying on this and treating experiments as immutable once they are running. It's important to note that the above only applies when changing the traffic split between variations. If you keep the split the same, but increase the percent of traffic included, users will not switch variations. For example, if you are running a 50/50 experiment on 20% of traffic, the variation ranges will be `0 to 0.1` and `0.5 to 0.6`. Users outside those ranges will be excluded from the experiment. If you increase the percent of traffic to 40%, but keep the 50/50 split, the ranges will become: `0 to 0.2` and `0.5 to 0.7`. As you can see, no users switch variations. Instead, some users who were previously excluded are now part of the experiment. ### What do I use for an "id" attribute in the SDK if my users aren't logged in? If your application has both logged-in and anonymous users, we recommend using two identifier attributes: - `id` which is the database identifier of logged-in users (or empty string for anonymous) - `deviceId` (or `sessionId`, etc.) which is a random anonymous hash, persisted in a cookie or local storage. This should always be set for both anonymous and logged-in users. If your application only has anonymous users (e.g. a static marketing site), then we recommend a single `id` attribute which, similar to `deviceId` or `sessionId` above, is a random hash persisted in a cookie or local storage. ## Running Experiments ### How do I run an A/B test in GrowthBook? The recommended way to run an A/B test is by using Feature Flags and our SDKs. 1. Create a feature in GrowthBook (e.g. `new-signup-form`) with an A/B Experiment rule 2. Use our SDKs to serve the different variations ```ts if (growthbook.feature("new-signup-form").on) { // Variation } else { // Control } ``` ### What is the best way to redirect users to a URL based on their experiment variation? You can now easily set up URL Redirect experiments within GrowthBook and customize navigation depending on your application. Read more about running a URL Redirect experiment [here](/app/url-redirects). ### How much traffic do I need to run A/B tests? What matters most for A/B testing is not traffic, but conversions. The general rule of thumb is to have at least 100-200 conversions per variation before you might start reaching significance. So that means if you do 50 orders per week and that's the metric you are trying to optimize, you'll need to run a simple 2-way A/B test for at least 4-8 weeks. If you run a 3-way test, it will take 6-12 weeks. ### Can I run multiple A/B tests at a time? Yes! In fact, we recommend running many experiments in parallel in your application. Most A/B tests fail, so the more shots-on-goal you take, the more likely you are to get a winner. Running tests in parallel is a great way to increase your velocity. Now it's possible your experiments might have interaction effects, but these are actually pretty rare in practice. One example is if one test is changing the text color on a page and another test is changing the background color. Some users might end up seeing black text on a black background, which is obviously not ideal. For these rare cases, you can use [Namespaces](/features/rules#namespaces) to run mutually exclusive experiments. Make sure all experiments within the same Namespace are using the same hash attribute (assignment attribute). As long as you apply a little common sense to avoid situations like the above, running multiple experiments has low risk and really high reward. ### How does GrowthBook handle event data? Check out the video below for a quick overview of how GrowthBook handles event data and how to fix common issues. ## GrowthBook Cloud ### What are the GrowthBook Cloud CDN usage limits? GrowthBook Cloud CDN has limits that depend on your plan. New free accounts can make up to 1 million CDN requests per month and consume up to 5GB of bandwidth. New Pro accounts include 2 million CDN requests and 20GB of bandwidth, and can pay for additional usage above this. Enterprise accounts have custom limits and volume discounts. For early adopters that joined GrowthBook Cloud before these new limits were introduced (March 2025), you will be grandfathered into the old limits of 10 million CDN requests and unlimited bandwidth per month. Thank you for being an early supporter! You can view your current usage at any time in the GrowthBook Cloud dashboard under [**Settings** > **Usage**](https://app.growthbook.io/settings/usage). ### How can I reduce my GrowthBook Cloud CDN usage? There are a few ways to reduce your GrowthBook Cloud CDN usage: - **SDK Caching** - Most of our [SDKs](/lib) support caching CDN responses. For back-end SDKs, this can drastically reduce the number of CDN requests since the response can be shared between many users. For front-end SDKs, caching has a smaller benefit, but can still help by eliminating duplicate requests from the same user. - **Nested CDNs** - You can add [your own custom CDN](/self-host/cdn) in front of our GrowthBook Cloud CDN. This will heavily decrease the number of requests that reach our CDN, but you will have to maintain this new infrastructure and pay for it outside of GrowthBook, so it might not end up reducing costs. - **GrowthBook Proxy** - You can [run self-hosted GrowthBook Proxy servers](/self-host/proxy) that cache CDN responses in your infrastructure. Like with nested CDNs, you will be responsible for maintaining and paying for this infrastructure. ### What happens if I exceed the limits on GrowthBook Cloud? We reserve the right to throttle or block CDN requests for accounts that exceed their free usage limits. We will always try to reach out to you before taking any action, but we reserve the right to take immediate action if we believe your usage is causing harm to other customers or our infrastructure. For Pro and Enterprise customers, we will work with you to find a solution that meets your needs. This could include increasing your limits, optimizing your usage, or moving to a custom plan. ## Self-Hosting ### Which docker image tag should I use when self-hosting? We recommend using the `latest` tag for both dev and production self-hosted deployments. This tag represents the latest stable build of GrowthBook and is what the Cloud app uses. Specific version tags (e.g. `v1.1.0`) are only released periodically (about once a month) and you will miss out on the many bug fixes and features added between major releases. We also recommend updating the image regularly. You can do that by downloading the latest image (`docker pull growthbook/growthbook:latest`) and restarting the container. ### What are the hardware requirements for self-hosting GrowthBook? The GrowthBook application is very lightweight and efficient. For most usecases, 2GB of memory is sufficient even for large production deployments. GrowthBook only deals with aggregate data and the bulk of the processing is offloaded to your data source. Because of this, you can easily analyze terrabytes of data from your laptop or a small container in the cloud. If you are using feature flags, **we strongly recommend** adding a caching layer between the GrowthBook API and your application in production. This will also help you stay within the limits of our [Fair Use Policy](https://www.growthbook.io/fair-use). Some of our [code examples](https://github.com/growthbook/examples) implement caching. We offer a pre-built [GrowthBook Proxy server](/self-host/proxy) you can run that handles caching and invalidation automatically. You can also setup [your own CDN](/self-host/cdn) or distributed cache like Redis. ### I can't upload to S3/getting 400 error when uploading to S3? - Make sure you've correctly set the `S3_BUCKET` and `S3_REGION` environment variables - Enable bucket ACL and set ownership to Bucket owner preferred: [read more here](https://stackoverflow.com/questions/70333681/for-an-amazon-s3-bucket-deplolyent-from-guithub-how-do-i-fix-the-error-accesscon). - Make sure the S3 bucket is publically accessible - Make sure CORS settings are correct. Add your URLs to the AllowedOrigins array or set to "\*" ## Debugging the SDKs ### Why is the trackingCallback not firing in the SDK? The `trackingCallback` only fires when a user is included in an experiment. If you're expecting to be included and you're still not seeing the callback fire, it's likely for one of the following reasons: - You are missing the `hashAttribute` for the experiment. For example, when you are splitting users by "company", but the company attribute is empty. - The feature is disabled for the environment you are in (dev/prod) - The experiment has reduced coverage. For example, if it's only running for 10% of users and you are in the 90% that are excluded. - There is another feature rule that is taking precedence over the experiment. If you are using the JavaScript or React SDK in a browser environment, you can install the GrowthBook DevTools Browser Extension for [Chrome](https://chrome.google.com/webstore/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/) to help you debug. **Note**: To use the plugin, you will need to pass `enableDevMode: true` when creating your GrowthBook instance. ```ts const growthbook = new GrowthBook({ enableDevMode: true, }) ``` ### How do I use the DevTools browser extension? - Make sure you are using the React or JavaScript SDK - Install the [Chrome](https://chrome.google.com/webstore/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/) DevTools Browser Extension - Pass the `enableDevMode: true` option into the GrowthBook constructor ```ts const growthbook = new GrowthBook({ enableDevMode: true, }) ``` ### My features aren't refreshing as expected. What can I do? Our SDK's implement a stale-while-revalidate approach to caching with a configurable time-to-live (TTL) value. This means that if the feature payload is considered stale (i.e. more than the TTL amount of time has passed since it's been updated), **the next request will return the stale features** and refetch an update asynchronously so that on the next request, the features will be up to date. You can learn more about how our SDK's implement this in detail [here](/lib/build-your-own#fetching-and-caching-features). If you would like something more real-time than this stale-while-revalidate approach, you may want to consider implementing the [GrowthBook Proxy](/self-host/proxy) on your self-hosted instance. ### How do I configure environments in the SDK? When you create an SDK connection, it is linked to a specific environment. [Learn more about environments](/features/environments). ### How do I make my own identifier? There are cases when using feature flags client side where the 3rd party identifiers used for assignment will be slow to load, and may cause flickering as some of the DOM rerenders. In these cases, generating your own identifier will make sure that features are assigned correctly when GrowthBook loads. This id that is generated will typically align one to one with the other identifiers, and does not need to be passed outside the SDK (though can be useful for debugging to pass this value in the trackingCallback). The code below can be used to generate a unique user id and save it in a cookie for the maximum amount of time allowed. Note, this technique is already included with our HTML/no-code SDK. Please be aware of any cookie policies this code may impact. This id will be unique to the browser and not the user, so if a user switches devices, they will have a different id. ```js const getUUID = () => { const COOKIE_NAME = "gbuuid"; const COOKIE_DAYS = 400; // 400 days is the max cookie duration for chrome // use the browsers crypto.randomUUID if set const genUUID = () => { if(window?.crypto?.randomUUID) return window.crypto.randomUUID(); return ([1e7]+-1e3+-4e3+-8e3+-1e11).replace(/[018]/g, c => (c ^ crypto.getRandomValues(new Uint8Array(1))[0] & 15 >> c / 4).toString(16) ); } const getCookie = (name) => { let value = `; ${document.cookie}`; let parts = value.split(`; ${name}=`); if (parts.length === 2) { let existing = parts.pop().split(';').shift(); setCookie(name, existing); return existing; } } const setCookie = (name, value) => { var d = new Date(); d.setTime(d.getTime() + 24*60*60*1000*COOKIE_DAYS); document.cookie = name + "=" + value + ";path=/;expires=" + d.toGMTString(); } // get the existing UUID from cookie if set, otherwise create one and store it in the cookie let existing = getCookie(COOKIE_NAME); if(existing) return existing; const uuid = genUUID(); setCookie(COOKIE_NAME, uuid); return uuid; } ``` Below is the same code, minified: ```js const getUUID=()=>{const a=(a,b)=>{var c=new Date;c.setTime(c.getTime()+86400000*400),document.cookie=a+"="+b+";path=/;expires="+c.toGMTString()};let b=(b=>{let c=`; ${document.cookie}`,d=c.split(`; ${b}=`);if(2===d.length){let c=d.pop().split(";").shift();return a(b,c),c}})("gbuuid");if(b)return b;const c=(()=>window?.crypto?.randomUUID?window.crypto.randomUUID():"10000000-1000-4000-8000-100000000000".replace(/[018]/g,a=>(a^crypto.getRandomValues(new Uint8Array(1))[0]&15>>a/4).toString(16)))();return a("gbuuid",c),c}; ``` ## Experiment Analysis ### What is the difference between a Dimension and a Segment? :::caution Segments are deprecated Segments have been deprecated and are being phased out. If you don't already have a Segment configured, you shouldn't create one — use a Dimension instead, or apply a custom SQL filter in the experiment's analysis settings. ::: A dimension is a user attribute that can have multiple values. Some examples are `country`, `account_type`, and `browser`. A segment was a specific group of users, defined with SQL from the "Data and Metrics → Segments" page. Some examples are `visitors in the US`, `premium users`, and `chrome users`. Segments could apply a filter to results, usually to compensate for bad data — for example, if your experiment was only visible to premium users, but your database inaccurately shows that free users were also included, you could apply a `premium users` segment to only include those who were actually exposed to the test. Dimensions are used to explore experiment results. For example, you can use a `country` dimension to see which countries had the highest conversion rates. Or an `account_type` dimension to see if there was a significant difference in how free vs paid users behaved. Or a `browser` dimension to detect any browser-specific bugs in your implementation. If you were relying on a Segment to filter out bad data (rather than to explore results), a custom SQL filter in the experiment's analysis settings is the recommended replacement, provided the dimension you need to filter on already exists on your experiment assignment query. ### How do I break down experiment results by more than one dimension? GrowthBook only lets you select a single dimension at a time when viewing experiment results. There isn't native support yet for combining two or more dimensions in one breakdown, but there are two workarounds: - **Custom SQL filter**: If one of the dimensions you care about already exists on your experiment assignment query, add a custom SQL filter in the experiment's analysis settings to restrict results to a specific value of that dimension, then use the dimension selector to break down by the second dimension. - **Custom dimension SQL**: Create a new dimension whose SQL computes the intersection of the two dimensions you care about (for example, concatenating `country` and `account_type` into a single value like `US - premium`). See [Dimensions](/app/dimensions) for how to write custom dimension SQL. ### My old exported notebook stopped working. How can I fix it? There's a good chance that the SQL we are exporting and your version of our Python stats library, `gbstats`, are out of sync. In February of 2023 we updated our SQL engines and `gbstats` library to only use sums and sums of squares, rather than averages and standard deviations. If the queries in your notebook return averages and standard deviations (using `AVG` and `VAR` SQL operators as part of the `__stats` CTE), then you need to run that notebook with `gbstats` version 0.3.1. You can [download this from PyPI](https://pypi.org/project/gbstats/0.3.1/) using `pip install gbstats==0.3.1` and ensure that your kernel uses that version of `gbstats`. Ideally in this case you can redownload the notebook and use the new `gbstats` library (0.4.0 or newer). You can download a new notebook by navigating to your experiment in GrowthBook and clicking `Download Notebook` again. This should now use the updated SQL and `gbstats` syntax. Then, if you install `gbstats` 0.4.0 or later, everything should work as expected. If the queries in your notebook return sums and sums of squares as part of the `__stats` CTE but your notebook is still erroring, then you probably have an old `gbstats` version installed and need to update to 0.4.0 or later. You can [download this from PyPI](https://pypi.org/project/gbstats) using `pip install gbstats` and ensure that your kernel uses that version of `gbstats`. ### How do I update the refresh rate for experiment data? Go to **Settings** → **General** → **Experiment Settings**. Change the **Experiment Auto-Update Frequency** field to the desired values. Save your changes. ![Setting to update experiment data refresh](/images/experiment-refresh.png) ### Are users counted multiple times in the experiment results? Deduplication ensures that each user is included only once in an experiment, even if they meet the eligibility criteria multiple times. This helps maintain accurate results and preserves data integrity. GrowthBook achieves deduplication by assigning a unique identifier (such as a `userID` or `sessionID`) to each user. When analyzing experiment data, it records only the first instance of a user’s exposure, preventing duplicate attributions. If a user participates in an experiment more than once (e.g., revisits the site multiple times), GrowthBook will count only their first recorded exposure. ## Other ### What types of SQL queries can users run in GrowthBook? Users can only run read SELECT queries in GrowthBook. For security reasons, all user-provided SQL is validated to be a read-only query before being executed. This applies to all places where you can write raw SQL, including: - SQL Explorer for ad-hoc queries - Metric and Fact Table definitions - Dimension queries - Data Source configuration (experiment assignment queries, identifier joins, etc.) Write operations such as INSERT, UPDATE, DELETE, etc, are not permitted from any user-entered SQL. :::note Pipeline Mode writes to your data warehouse Some GrowthBook features, such as [Pipeline Mode](/app/data-pipeline), may write temporary tables to your data warehouse to improve query performance. However, these write operations are managed by GrowthBook itself. ::: ### How do I disable the on-screen celebrations? Throughout the GrowthBook application, we randomly celebrate key milestones like launching experiments with on-screen confetti. If you'd like to disable this, you can click on your avatar in the top right corner and select "Edit Profile". From there, you can disable the toggle for "Allow Celebrations". Please note this is persisted in your browser's local storage, so if you clear your browser's local storage, you will need to disable this again. --- ## Can't find your question? If you can't find an answer to your question above, please let us know so we can help you out and improve the docs for future users! You can join our [Slack channel](https://slack.growthbook.io?ref=docs-faq) for the fastest response times. Or send an email to hello@growthbook.io if Slack isn't your thing. --- ## Feature Flag Fundamentals import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; This page covers the core concepts you'll need to create and configure feature flags in GrowthBook. ## Feature Keys Every feature is defined by a unique **key**. This is what your engineering team references in code when checking the value of a feature. Feature keys **cannot** be changed after creation, so take care when choosing one. If you make a mistake, delete the feature and create a new one. ![Create Feature modal in GrowthBook](/images/features/feature-create-feature-2.png) :::warning Naming Conventions Feature keys must only include letters, numbers, hyphens (-), and underscores (\_). ::: ## Feature State Feature flags can be enabled or disabled. When a flag is disabled, GrowthBook excludes the feature from the API response. The feature evaluates to `null` and ignores all targeting and rules. In your app, the SDK will render the feature with the fallback value. Here's an example from the JavaScript SDK: ```js const showNewCheckout = gb.isOn('new-checkout') // Renders false if the feature is disabled const buttonColor = gb.getFeatureValue('button-color', 'red') // Renders 'red' if the feature is disabled ``` ## Feature Types Features can be a simple ON/OFF flag or a more complex data type (string, number, or JSON). | Type | Values | Best for | Example | | ----------- | -------------------- | ------------------------------------------------------ | --------------------------------------------- | | **Boolean** | `true` / `false` | Simple on/off toggles, kill switches, gradual rollouts | `show-onboarding-checklist` | | **String** | Any string | Remote config, multivariate A/B tests | `button-color` set to `"red"`, `"blue"`, etc. | | **Number** | Any integer or float | Numeric configuration values | `results-per-page` set to `10`, `25`, `50` | | **JSON** | Any JSON object | Complex configuration, multiple values in one flag | `{"maxRetries": 3, "timeout": 5000}` | **Boolean** flags are the most common. They're limited to two variations in A/B tests (on/off), so if you need more variations, like testing three different button colors, use a string, number, or JSON flag instead. **JSON** flags also support JSON Schema validation, which lets you enforce the structure of the value before it reaches your application. A **JSON** flag can also be **backed by a [Config](/features/configs)** — a reusable, typed, composable object shape you define once and enforce on every flag that uses it. Reach for one when many flags share the same structured value; changing the config updates them all. See [Configs](/features/configs) for details. **Reusable values.** A **string** or **JSON** flag value can reference a [**Constant**](/features/constants) — a named value (with optional per-environment overrides) defined once and resolved into the value when the SDK payload is built. See [Constants](/features/constants). ## Default Values Each feature has a default value that's used when there are no matching rules for the user. A feature that is enabled on an environment with no rules uses the default value. Default values match the feature type. Default values can be overridden by [**Rules**](/features/rules), which let you target specific users, roll out to a percentage of traffic, or run A/B experiments. ## What's Next Now that you understand keys, types, and default values, the next concepts to explore are: - **[Environments](/features/environments)** — Control where a feature is active (e.g., enabled in dev but not production). Each environment has its own SDK connection and independent set of rules. - **[Rules](/features/rules)** — Override default values by targeting specific users, rolling out to a percentage of traffic, or running A/B experiments. See [Choosing a Rule Type](/features/rules#choosing-a-rule-type) for help picking the right approach. - **[Targeting Conditions](/features/targeting)** — Define _who_ sees a rule using attributes like user ID, country, or plan type. --- ## Code References import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Code References Code References lets you see exactly where feature flags are used in your codebase. A CLI tool scans your repository and sends the results to GrowthBook, which then surfaces this information on each feature's page. This helps you find the exact lines of code that reference a flag — useful when cleaning up stale flags or understanding a flag's scope. ![Code References](/images/features/features-code-refs.png) :::tip Code References also improves [Stale Feature Detection](/features/stale-detection) accuracy by providing concrete evidence of whether a flag is still referenced in code. ::: ## Getting Started For GitHub users, a streamlined GitHub Action integrates with code references: - [GrowthBook Code References GitHub Action](https://github.com/marketplace/actions/growthbook-code-references) For other platforms, use the CLI tool or Docker image: - [GrowthBook gb-find-code-refs CLI utility](https://github.com/growthbook/gb-find-code-refs) - [Dockerized version of gb-find-code-refs](https://hub.docker.com/r/growthbook/gb-find-code-refs) ## Enabling Code References in GrowthBook To enable code references, navigate to your General Settings page and scroll to **Configure Code References**. ### Branch Filtering Specify which branch names (comma-separated) to show code references for. By default, code references from all branches are displayed. For example, if your team only needs references from `main` and `qa`, set this option to `main, qa`. ### Platform Links GrowthBook can link directly to your codebase on GitHub or GitLab. Select your platform in the settings and code references will automatically include links to the relevant files and lines. If your platform isn't supported, let us know in the [Slack community](https://slack.growthbook.io?ref=coderefs). ## Viewing Code References Once enabled, you can view Code References on a specific feature page, under the Code Refs tab. --- ## Configs import CommercialFeature from '@site/src/components/CommercialFeature'; Configs are **typed, validated, composable objects** you define once and reuse across feature flags. Where a [constant](/features/constants) is a reusable _value_, a config is a reusable _shape_: it carries both a value **and** a schema, so every flag (and every child config) built on it is validated against that schema. The main reasons to reach for a config: - **Type safety** — a config defines the fields it contains and their types. Anything that uses the config is checked against that definition. - **Validation** — values that don't match the schema are rejected at edit time, before they ever reach an SDK payload. - **Composability** — a config can extend another config, layering fields on top of an inherited base. You can then attach a config to a feature flag as the flag's value, so the config's shape is **enforced** on that flag. Secondary benefits: - **Import/export of values** — pull a config's value in or out as JSON. - **Import/export of type definitions** — pull a config's schema in or out as [JSON Schema](https://json-schema.org/) or TypeScript. ## Anatomy of a config A config has: - A **key** — a slug auto-generated from the name (e.g. "Checkout Flow" → `checkout-flow`), used to reference it. - A **value** — a JSON object. - A **schema** — the field definitions (names, types, required/optional, bounds, enums, nullability) the value is validated against. - An optional **parent** — another config's key. The config inherits the parent's fields and value, then layers its own on top. - Optional **environment overrides** — per-environment (and optionally per-project) patches layered on top of the base value, covered in [Environment overrides](#environment-overrides). A config's value applies across all environments by default. When you need it to differ per environment, add [environment overrides](#environment-overrides); you can also draw an environment-varying value from a [constant](/features/constants). ## Type safety and validation A config's **schema** is the source of truth for its shape. The editor builds it for you as you define fields, and validates the value against it on every save — a missing required field, a wrong type, an out-of-range number, or a value outside an allowed enum is rejected before it can be saved. By default a config is **closed** (extra keys not in the schema are rejected). A config can opt in to being **extensible** (allow extra keys), and your organization has a default for new configs. Even when extensible, the typed fields are still validated. ## Composability and inheritance A config can name another config as its **parent**. The child inherits the parent's fields and value and layers its own on top — like extending a base type. Inheritance is resolved when the SDK payload is built, so children always reflect the latest parent. Precedence, lowest to highest: 1. The parent chain, root first (each ancestor's value, then schema) 2. The config's own value and schema When fields collide, the **closest base wins**: a child cannot redefine a field that a published ancestor already owns. This keeps a family's shape coherent — the base config governs the fields it declares, and children may only _add_ fields. Changing a base cascades down to its descendants automatically. Values layer as a **deep (targeted) patch**: when a child and its inherited value are both objects, they merge **recursively, key by key**, so a child restates only the leaves it changes and inherits the rest of a nested object. **Arrays and scalars replace wholesale** (arrays are never merged element-by-element), and **`null` is a value** (it sets the field to `null`, it does not delete it). A value composed from a [constant](/features/constants) via `$extends` is applied **whole** — the patch never reaches inside a referenced chunk. [Constants](/features/constants) compose the opposite way: a constant's own keys **replace** what its `$extends` bases provide, wholesale — configs and feature values patch; constants are atomic building blocks. A config's value can also interpolate [string constants](/features/constants) with `{{ @const:key }}` inside any string field, exactly as in feature values — including constants with per-environment overrides, another way to vary a config's value by environment alongside [environment overrides](#environment-overrides). You can view the full family tree (ancestors and descendants) for any config, both in the app and over the [REST API](#rest-api) via the lineage endpoint. ## Environment overrides By default a config resolves to a single **base value** used in every environment. When a value needs to differ by environment, add an **environment override** — a patch, scoped to one or more environments (and optionally projects), that layers on top of the base for those environments only. Manage overrides from the environment tabs on the config's page: **All Environments** shows the base value, and each environment that has an override gets its own tab for editing just that environment's patch. A [config-backed feature flag](#using-a-config-as-a-feature-flags-value) then serves the base value where no override applies and the overridden value where one does. An override resolves as the **last layer**: the matching environment's patch is deep-merged onto the fully-resolved base — the same targeted, key-by-key merge as [inheritance](#composability-and-inheritance) — so it wins over inherited and composed values while the base still cascades for every key it doesn't touch. When more than one override could match a given environment and project, the **first match wins**. Each override is itself a config, so its value carries its own revisions and goes through the same draft → review → publish flow; adding or removing an override for an environment takes effect immediately. ## How configs are validated A config's value passes through several layers of validation before it reaches an SDK payload: 1. **Schema conformance** — each field is checked against its schema definition (types, required fields, enums, bounds, nullability, and whether extra keys are allowed). This is the one-field-at-a-time check described in [Type safety and validation](#type-safety-and-validation). 2. **Cross-field validation rules (invariants)** — mongrule conditions that express constraints across fields, which a per-field schema can't. Covered just below. 3. **Custom validation hooks** — optional customer-authored JavaScript that runs on save and publish. See [Custom validation hooks](#custom-validation-hooks). 4. **Descendant enforcement on publish** — publishing a config re-checks every descendant against its effective rules. See [Where rules run](#where-rules-run). The sections below cover each layer in turn. ## Validation rules (invariants) Schema fields validate one field at a time. **Validation rules** (called `invariants` in the API) express the cross-field constraints a schema can't — "target must not exceed max", "if A is set, B must be too". Each rule has a **name**, the **rule expression**, and a human-readable **message** shown when it fails. Rules are part of the config's schema — edit them alongside the fields in the app, or write the schema's `invariants` array over the REST API. Rules are inherited: a config is checked against its ancestors' rules plus its own, accumulated base → leaf, and a rule with the same name as an ancestor's overrides it. That means a base config's rules guard every descendant. ### Rule format A rule is a **MongoDB-style condition** — the same condition syntax as feature and experiment targeting rules, so it's a familiar, standard GrowthBook concept — extended with `{ "$ref": "path" }` to compare against another field of the value instead of a literal: ```json { "buffer.target_seconds": { "$lte": { "$ref": "buffer.max_seconds" } } } ``` This is the one format everywhere: it's what you author in the UI, what's stored on the schema, what's evaluated at publish, and what the REST API accepts on write and returns on read. The full condition operator set is available (`$eq`, `$ne`, `$lt`/`$lte`/`$gt`/`$gte`, `$exists`, `$and`/`$or`/`$not`/`$nor`, …). **Nested fields are supported.** A dot-separated path like `buffer.target_seconds` reaches into nested objects — as the tested field and as a `$ref` target alike. A path that doesn't exist evaluates as `null` rather than erroring. ### Where rules run Rules are enforced on the GrowthBook server **when a config publishes** — on every publish path (direct API writes, publishing a revision, scheduled and auto-publish) — and are evaluated against the **fully-resolved** value (own keys plus everything inherited). They never run in SDKs; payloads carry resolved values only. While editing, the schema editor shows live pass/fail per rule, but a draft can be saved in a violating state — only publishing enforces. Because publishing a config changes every descendant's resolved value, the publish also re-checks each descendant against its effective rules. Only violations **introduced by that publish** block it: a pre-existing violation elsewhere in the family never blocks an unrelated edit. The **Block publishing on JSON schema errors** organization setting controls what happens when a value fails its schema or a validation rule, or when a schema change makes a dependent Config or Feature Flag invalid: - **On (default)** — publishing is blocked. A REST API caller with **Bypass draft approvals** in every Project can send `"skipSchemaValidation": true` to continue. - **Off** — GrowthBook returns a warning instead. Confirm the warning in the app, or send `"ignoreWarnings": true` through the REST API. The editor shows these problems on the affected fields before publishing. This setting does not apply to Custom Hook errors. ## Custom validation hooks You can add your own validation with **Custom Hooks**, which are sandboxed JavaScript functions that run when a Config is saved or published. `validateConfig` runs when a Config is created or updated, and `validateConfigRevision` runs whenever a revision is published. A hook can return a warning or reject the change. Anyone can acknowledge a warning with `"ignoreWarnings": true`. To publish after a rejection, a REST API caller with **Bypass draft approvals** in every Project must send `"skipHooks": true`. The `"skipSchemaValidation"` field does not bypass hooks. Hooks are scoped Global, Project, or Config; a config-scoped hook automatically covers the whole family (the scoped config and everything that inherits from it). Manage them from a config's **Validation** tab or under **Settings → Custom Hooks**. :::note Custom Hooks are only available on self-hosted GrowthBook Enterprise. ::: For the full hook API, examples, limits, and warning behavior, see [Custom Hooks](/features/custom-hooks). ## Using a config as a feature flag's value This is where configs pay off: attach a config to a JSON feature flag — as the flag's **default value** or on a specific **rule** — and the config's shape is enforced on that flag. The flag serves the resolved config value, and edits to the flag are validated against the config's schema. Any rule type can serve a config-backed value, including **experiment** and **contextual-bandit** rules, where each variation carries its own override. Because the config's schema is authoritative, a flag can't also define its own JSON schema while its value is backed by a config — detach the config or remove the flag's schema. The flag's value is stored as a small **override patch** layered on top of the config, so changing the config updates every flag that uses it. ## Seeing where a config is used Every config tracks where its keys are consumed. On the config's page — and over the [REST API](#rest-api) — GrowthBook lists each feature-flag default value, rule, experiment, and contextual bandit that overrides one of the config's keys (across live and draft revisions), with the exact value each reference sets. Per-key badges surface the count at a glance, and for experiment and bandit rules you can inspect the override each variation applies. ## Protecting keys used by running experiments Changing a config value that a **running experiment** depends on can distort that experiment's results. You can opt a config into a guard — an organization-wide default, overridable per config — that **blocks publishing** a change to any key currently used by a running experiment or contextual bandit. The guard is computed live, so it clears on its own once the experiment stops; there's nothing to unlock. To publish anyway, confirm the warning in the app or send `"ignoreWarnings": true` in the REST API request. Disabling the guard for a Config requires **Bypass draft approvals** access. ## Import and export A config's value and its type definition can move in and out of GrowthBook: - **Value** — import or export the config's JSON value. - **Schema** — import or export the type definition as **JSON Schema**, **TypeScript**, or another supported typed-code format (Protobuf, Python, Go, Rust), or infer a schema from an existing value. JSON Schema is the canonical interchange format. :::note The TypeScript option is a feature-limited convenience The TypeScript import/export is a convenience for round-tripping simple `type`/`interface` shapes — it is **not** a full TypeScript language parser. It understands the common building blocks (primitive fields, objects, arrays, simple unions/enums, optional and nullable fields) and ignores or rejects anything beyond that (generics, conditional/mapped types, imported types, functions, etc.). For anything non-trivial, use JSON Schema as the source of truth. ::: ## Keeping code and configs in sync at scale Configs are designed to be the **source of truth** for the configuration they describe. Rather than hand-maintaining the same shape in many codebases — where copies drift apart over time — define the config once in GrowthBook and have your services consume it. The schema and value live in one place; code reads them. A typical end-to-end workflow: 1. **Define the config in GrowthBook** — author it in the UI, or import it once from an existing code definition (see [Import and export](#import-and-export)). 2. **Generate types from the config.** Export the schema as TypeScript (or JSON Schema) and commit the generated types so your code is type-checked against the config's current shape. Regenerate when the config changes — the same pattern teams use for generated API clients or protobuf types. 3. **Consume the resolved value at runtime** through a JSON feature flag backed by the config (see [Using a config as a feature flag's value](#using-a-config-as-a-feature-flags-value)). The SDK serves the resolved value; the generated types describe its shape. 4. **Guard against drift in CI.** The schema **verify** endpoint checks whether the shape your code expects still matches the config in GrowthBook. It reports whether the two are in sync and, when they differ, a **categorized diff** — _contract_ changes (types, required-ness, enums, structure) separated from _documentation-only_ changes (descriptions) — so a CI step can fail the build on real contract drift while treating doc changes as informational. Because types are compile-time and values are served at runtime, follow the same forward/backward-compatibility hygiene as any API contract: ship new fields with **defaults** (so a valid value exists the moment the field appears), and **deprecate before removing**. ### Named projections (per-source type names) When you import a config's schema from TypeScript and supply a **source** identifier (typically a service or codebase name), GrowthBook captures that source's named-type structure — for example, that the object at `http.retry` was declared as `RetryPolicy`. Exporting the schema with the same `source` reproduces those names instead of inlining the shapes: - **Without a source** — nested objects are inlined: `retry: { maxAttempts: number; … }`. - **With a source** — named interfaces are emitted: `interface RetryPolicy { … }` and `retry: RetryPolicy`. Projections are rendered against the config's **current** schema, so the captured names stay attached as the config evolves — an export never serves a stale snapshot. Different sources can each keep their own type names for the same config. Projections are presentation metadata only: they never affect validation, the canonical schema, or drift detection. ## Editing and approvals Config changes go through the same draft → review → publish flow as features, governed by your organization's [require-reviews settings](/features/publishing-and-approval-flows) matched on the config's project. You can view past versions, compare revisions, and revert from the config's page. Some guardrails: - **Archiving** is blocked while a config is still referenced or while it has live child configs — resolve those first. - **Deleting** is blocked while other configs still inherit from it (it would dangle their parent link); delete or re-parent the children first. - **Cycles** in lineage or value references are rejected at save time. ## REST API Configs and their revisions are fully manageable over the [REST API](/api/#tag/configs) — create, update, archive, delete, list references, fetch the full lineage, see where each key is used, manage [environment overrides](#environment-overrides), import/export value and schema, and drive the revision/approval flow programmatically. Webhook events (`config.*`, including `config.revision.*`) fire on every change, modeled after the feature and constant events. To get the override values behind a key's usage, follow each reference back to its feature (`GET /features/:id`) — the key-usage endpoint reports _where_ a key is used, and the feature is the source of truth for _what_ it's set to. ## Permissions Creating, editing, and deleting configs requires the **Manage Configs** permission (the `ConfigsFullAccess` policy), which is included in the Engineer, Experimenter, and Admin roles by default. Configs can be scoped to specific projects. --- ## Constants import MaxWidthImage from "@site/src/components/MaxWidthImage"; Constants are reusable named values you define once in GrowthBook and reference from feature flag values. Change a constant in one place and every flag that references it picks up the new value the next time its SDK payload is built — no need to edit each flag. Common uses: - A shared endpoint or hostname used across many flags (e.g. an API server URL). - A block of JSON config reused by several flags. - Values that differ per environment (e.g. a production vs. staging URL) behind a single reference. ## Constant types Each constant has one type: - **String** — a plain string value (URLs, keys, identifiers, etc.). - **JSON** — a JSON **object** (key/value map), used as a reusable config template merged via `$extends`. ## Per-environment values A constant has a base **value** plus optional **per-environment overrides**: when a flag is built for an environment, its override is used if set, otherwise the base value. A reference to a constant with no value for the target environment is left as-is in the payload. ![Editing a constant value with per-environment overrides](/images/features/constant-edit.png) ## Referencing constants Constants are referenced by their **key** — a slug auto-generated from the name (e.g. a constant named "API Server" gets the key `api-server`). In a feature value editor, use **Insert constant** to pick one, or type the reference by hand. The reference syntax depends on the constant's type: ![Inserting a constant reference into a feature value](/images/features/constant-reference.png) ### String constants — `{{ @const:key }}` Interpolate a string constant anywhere inside a feature's or a [config](/features/configs)'s string value (including inside a string field of a JSON value): ``` Connect to {{ @const:api-server }} on port {{ @const:api-port }}. ``` ```json { "endpoint": "{{ @const:api-server }}" } ``` ### JSON constants — `$extends` JSON constants are objects (key/value templates). Compose them with an `$extends` array that lists one or more constant references. Each referenced object is merged in, in order, and the object's own keys override the merged result: ```json { "$extends": ["@const:default-config"], "timeout": 30 } ``` `$extends` works for a nested value too: ```json { "limits": { "$extends": ["@const:default-limits"] } } ``` Merge precedence, lowest to highest: 1. `$extends` references, in array order (later references override earlier) 2. the object's own keys So the position of `$extends` in the object doesn't matter — own keys always win — but the order of references _within_ the array does. **Own keys replace wholesale.** A constant is an atomic building block: whatever an own key states is exactly what it resolves to. When an own key and the merged base are both objects, the own value **replaces** the inherited one outright — it is not merged into it. For example, extending a base of `{ "retry": { "connect": 1000, "read": 5000 } }` with own keys `{ "retry": { "read": 8000 } }` resolves to `{ "retry": { "read": 8000 } }` — `connect` is dropped, not inherited. Restate any inherited fields you want to keep. (Writing `null` sets the value to `null`; it does not delete the key.) :::info Constants replace; configs and feature values patch This is deliberately the opposite of how [configs](/features/configs) and feature values compose. A child config — or a config-backed feature value — is a **deep, targeted patch**: it merges recursively, key by key, restating only the leaves it changes. A constant's own keys are authoritative wholesale. The same `$extends` syntax written directly in a feature or config value follows the deep-patch rule, not this one — the mode depends on which entity's keys they are, not on what is being extended. ::: A referenced chunk is always applied **whole**: an `$extends` reference (or a nested object that itself uses `$extends`) drops in as a complete unit. A constant's value can itself reference other constants; they're resolved recursively. :::note Advanced: control merge order with inline objects An `$extends` entry can also be an **inline object** instead of a `@const:` reference. It merges at that position, so a _later_ reference can override it — something own keys can't do (they always win). Most templates only need references plus own keys; reach for this only when you need a literal layer beneath a referenced one. ```json { "$extends": [{ "timeout": 30 }, "@const:override-pack"] } ``` Here `@const:override-pack` can override the inline `timeout`. Loose entries that are neither a reference nor an object (numbers, booleans, bare strings) are rejected when you save. ::: ## How resolution works References are resolved **when the SDK payload is built**, on the server. SDKs receive the fully-resolved value, so no SDK changes or upgrades are required. A few rules: - **On by default.** Feature and config values are scanned for references automatically. - **Escape literals.** To keep a literal `{{ @const:... }}` or `@const:` key in a value, wrap it in backticks. - **Graceful failure.** An unresolvable `{{ @const:... }}` interpolation — unknown constant, type mismatch, or a cycle — is left in place as literal text rather than failing the build. An unresolvable `$extends` reference is dropped instead (a leftover directive would be invalid config). - **Archived references are stripped.** Archiving a referenced constant is blocked (see [Editing and approvals](#editing-and-approvals)), but any reference that still points at an archived constant is removed from the value (string interpolations dropped, JSON references removed) rather than resolved. ## Protecting keys used by running experiments If a constant feeds a [config](/features/configs) whose running-experiment guard is enabled, publishing a change to the constant is subject to that same guard: it is blocked when the change would alter a config key currently used by a running experiment or contextual bandit. The guard is computed live, so it clears on its own once the experiment stops. To publish anyway, acknowledge the block — confirm the warning in the app, or pass `"ignoreWarnings": true` in the body of the REST publish request. See [Protecting keys used by running experiments](/features/configs#protecting-keys-used-by-running-experiments) for the config-side details. ## Editing and approvals Constant changes go through the same draft → review → publish flow as features, governed by your organization's [require-reviews settings](/features/publishing-and-approval-flows) matched on the constant's project. You can view past versions, compare revisions, and revert from the constant's page. Archiving is blocked while a constant is still referenced — the archive dialog lists every feature and constant that uses it, so you can remove those references first. ## REST API Constants are fully manageable over the [REST API](/api/#tag/constants) — create, update, archive, list references, and drive the revision/approval flow programmatically. ## Permissions Creating, editing, and deleting constants requires the **Manage Constants** permission (the `ConstantsFullAccess` policy), which is included in the Engineer, Experimenter, and Admin roles by default. Constants can be scoped to specific projects. --- ## Custom Hooks # Custom Hooks With self-hosted GrowthBook Enterprise, you can extend GrowthBook's validation logic with Custom Hooks. Custom Hooks are JavaScript snippets that run on the server during validation. Use them to enforce naming conventions for feature flags, check for required metadata, or add other custom validation logic. ## Using Custom Hooks :::note Custom Hooks are only available on self-hosted GrowthBook Enterprise. ::: You can create multiple hooks of the same type for flexible, granular validation rules. Each hook has a **scope** that controls which resources it runs for: - **Global** — runs for every feature or experiment (depending on hook type). - **Project** — runs only for resources in the selected projects. - **Feature** — runs only for a single feature. Manage global and Project-scoped hooks under **Settings → Custom Hooks**. Manage a hook for one Feature Flag from that flag's **Validation** tab. This tab also lists the global and Project hooks that apply to the Feature Flag. Experiment hooks can only use a global or Project scope. The same model applies to [Configs](/features/configs). The `validateConfig` and `validateConfigRevision` hook types run when a Config is saved or published and can use a global, Project, or Config scope. Manage Config-scoped hooks from the Config's **Validation** tab or from **Settings → Custom Hooks**. A Config-scoped hook also runs for every Config that inherits from it, including indirect descendants through `parent` or `extends`. If a publish moves a Config into that family, the hook runs during the same publish. A descendant Config's page lists both its own hooks and the hooks it inherits. Creating or changing a global, Project-scoped, or Config-scoped hook requires the **Custom Hooks** permission. A hook for one Feature Flag requires **Edit** access to that flag instead. The REST API can change a hook's scope after creation. The caller must have permission to manage both the old and new scopes. To remove an entity scope, send `entityType: null` and `entityId: null`. ### Limits Custom Hooks are executed in a V8 Isolate, which provides a secure and efficient environment for running untrusted code. All modern JavaScript language features are supported (including async/await and fetch), but certain global objects like `process.env` are not available for security reasons. The following default limits are in place to prevent abuse and ensure performance. They can all be tweaked via environment variables: - `CUSTOM_HOOK_MEMORY_MB` - Maximum memory allocation for the isolate (default: 32MB) - `CUSTOM_HOOK_CPU_TIMEOUT_MS` - Maximum active CPU time (default: 100ms) - `CUSTOM_HOOK_WALL_TIMEOUT_MS` - Maximum total run time (including async calls) (default: 5000ms) - `CUSTOM_HOOK_MAX_FETCH_RESP_SIZE` - Maximum response size from fetch calls in bytes (default: 500KB) ### Execution Frequency A hook may execute **multiple times for a single save**. GrowthBook runs hooks early in a request (before related records are written) and again immediately before the final database write, and the Incremental Changes option adds additional runs against the previous state. Keep hooks fast and free of side effects — they should validate their inputs and either return, `throw`, or `addWarning()`, nothing else. ### Debugging If a Custom Hook throws an error during execution, the error message will be used as the validation error shown in the UI. This allows you to provide clear feedback to users about why their changes were rejected. Hook errors are prefixed with `Custom hook:`, which distinguishes them from schema-conformance errors (prefixed with the offending field, e.g. `value: …`) and from validation-rule failures (which surface the rule's own message). When creating a Custom Hook, use the built-in test interface to tweak inputs and run the hook. The test output shows all errors, warnings, console messages, and the return value (if any). Use `console.log` statements liberally while developing hooks to inspect variables and understand the flow of execution. ### Warnings Instead of `throw`, a hook can call `addWarning("message")` to raise a **soft warning**. In the app, a user can review the warning and select **Save anyway**. A REST API client can send `"ignoreWarnings": true`. A thrown error blocks the change. `ignoreWarnings` does not bypass it. A REST API caller with **Bypass draft approvals** in every Project can send `"skipHooks": true` to continue despite the error. The `"skipSchemaValidation"` field does not bypass Custom Hooks. ```js if (feature.tags.length === 0) { addWarning("Consider adding at least one tag"); } ``` Because execution continues after `addWarning`, you can raise warnings and still `throw` later in the same hook. ### Incremental Changes Each Custom Hook has an **Incremental Changes Only** option that affects behavior during update operations. When enabled, the hook is skipped if the same error was already present before the update. A hook receives only the arguments listed for its type, and they always describe the proposed state. To act on what a change introduces, check that state and enable this option: GrowthBook re-runs the hook against the previous state and discards the error when both runs report the same message, so avoid interpolating changing values into a message you want suppressed. ```js // Blocks adding the "restricted" tag; features that already carry it are unaffected. if ((feature.tags || []).includes("restricted")) { throw new Error("The 'restricted' tag is reserved and cannot be added."); } ``` This is especially useful for enforcing rules that may be difficult to fix retroactively. For example, if you require all features to have at least one tag, enabling this option will prevent users from being blocked by existing features that violate this rule when they attempt to make unrelated changes. In some cases, you should NOT enable this option. For example, a hook that prevents publishing changes to "locked" features should run every time to ensure the lock is still in place. **Important**: When using this option, be sure to limit each hook to a single validation check. Otherwise, a failing check early in the hook code may cause later checks to be bypassed unintentionally. ## Hook Types GrowthBook supports several Custom Hook types. Each is triggered at a different point in the validation process and receives different input parameters. ### validateFeature Called whenever the feature itself is written: on create, and on publish, including an edit that publishes immediately. Changes staged into a draft are covered by `validateFeatureRevision` instead. Receives one argument, `feature`: the feature as it will look once the change lands. Example: Require a non-empty description before the feature is toggled ON in production. ```js if (feature.environmentSettings.production.enabled) { if (feature.description.trim() === "") { throw new Error( "Feature description is required when enabling in production." ); } } ``` Example: Require all features to have at least 1 tag. ```js if (feature.tags.length === 0) { throw new Error( "All features must have at least one tag." ); } ``` Example: Don't allow empty objects as the default value for JSON features. ```js if (feature.valueType === "json" && feature.defaultValue === "{}") { throw new Error( "Default value for JSON features cannot be an empty object." ); } ``` ### validateFeatureRevision Called whenever a feature revision is created or updated, including while a change is still staged in a draft. Receives two arguments: - `feature`: the live feature, as it stands before this revision publishes. - `revision`: the revision being written, including `version`, `baseVersion`, `status`, `comment`, `title`, `defaultValue`, `rules`, `createdBy`, `contributors`, `reviews`, and `metadata`. `revision.metadata` holds the feature-level fields the revision carries (`tags`, `description`, `owner`, `project`, `customFields`, `jsonSchema`, and more), captured when the draft was created and updated as further changes are staged. Publishing applies only the difference between the revision and its base, so a value here is not necessarily one this revision will change. Older revisions may omit fields, so read them with optional chaining. Example: Block staging a reserved tag. Enable **Incremental Changes Only** so features that already carry it aren't blocked on unrelated edits. ```js if ((revision.metadata?.tags || []).includes("restricted")) { throw new Error("The 'restricted' tag is reserved and cannot be added."); } ``` Example: Require a comment before publishing a draft. ```js if (revision.status === "published" && !revision.comment) { throw new Error( "A comment is required before publishing a revision." ); } ``` Example: Require all percentage rollouts to use `userId` as the hashing attribute. ```js for (const env in revision.rules) { for (const rule of revision.rules[env]) { if (rule.type === "rollout" && rule.hashAttribute !== "userId") { throw new Error( "All rollouts must use 'userId' as the hash attribute." ); } } } ``` Example: Don't allow targeting by PII (e.g. email address) ```js const piiAttributes = ["email", "phone", "ssn"]; for (const env in revision.rules) { for (const rule of revision.rules[env]) { if (rule.condition) { for (const attr of piiAttributes) { // `condition` is a stringified JSON object // Look for the quoted attribute name anywhere in the string if (rule.condition.includes(`"${attr}"`)) { throw new Error( `Targeting by PII (${attr}) is not allowed.` ); } } } } } ``` Example: Call an external service to validate feature naming conventions. ```js const response = await fetch( "https://example.com/validate-feature-name", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ featureName: feature.name }), } ); const result = await response.json(); if (!result.isValid) { throw new Error( result.message || "Feature name validation failed." ); } ``` Example: If a feature has a "locked" tag, prevent publishing changes (except for one specific admin). ```js if ( feature.tags.includes("locked") && revision.status === "published" ) { if ( revision.publishedBy?.email !== "admin@example.com" ) { throw new Error( "This feature is locked and cannot be published." ); } } ``` #### Enforcing approval policies at publish The hook runs at publish time with `revision.status === "published"` on the proposed revision, so you can gate publishes specifically. `revision.reviews` holds the active reviewer verdicts for the current review cycle — one entry per reviewer in the shape `{ userId, user, status, timestamp }`: - `userId` — stable reviewer identifier: the user ID for dashboard users, or the API key ID for service accounts. - `user` — the full event user: `{ type: "dashboard", id, name, email }` for humans, `{ type: "api_key", apiKey }` for service accounts (where `apiKey` is the key's ID, not its secret). - `status` — `"approved"` or `"changes-requested"` for active verdicts. When draft content changes after a verdict is given (and the org's review settings reset reviews on change), the verdict is demoted to `"approved-stale"` / `"changes-requested-stale"` — still attributable, but no longer an active verdict, so policies matching on the active statuses ignore it automatically. A verdict that was retracted no longer appears, and all verdicts clear when a new review cycle starts (review re-requested or recalled). - `timestamp` — when the verdict was submitted. Compare against `revision.dateUpdated` to detect approvals that predate later edits. Disable **Incremental Changes Only** for publish-gating hooks so the policy is enforced on every publish. Example: Require approval from a specific service account plus at least one human. ```js if (revision.status === "published") { const approvals = (revision.reviews || []).filter( (r) => r.status === "approved" ); if (!approvals.some((r) => r.user?.type === "api_key" && r.user.apiKey === "key_abc123")) { throw new Error( "Publishing requires approval from the release-bot service account." ); } if (!approvals.some((r) => r.user?.type === "dashboard")) { throw new Error( "Publishing requires at least one human approval." ); } } ``` Example: Require two approvals, at least one from a designated reviewer group. ```js const seniorReviewers = ["user_abc", "user_def", "user_ghi"]; if (revision.status === "published") { const approvals = (revision.reviews || []).filter( (r) => r.status === "approved" ); if (approvals.length seniorReviewers.includes(r.userId))) { throw new Error("At least one approval must come from a senior reviewer."); } } ``` ### validateConfig Called whenever a [config](/features/configs) is created or updated. Receives the config (its fields, staged `value`, a `lineage` object, `isHookTarget` / `hookTargetKey`, and — when the config is an [environment/project override](/features/configs#environment-overrides) — a `scopedConfig` object) as input. `config.value` is a parsed JSON object — read its keys directly, no `JSON.parse` needed. Since a config-scoped hook also runs for descendants, `config.isHookTarget` is `true` only for the exact config the hook is pinned to. When present, `config.scopedConfig` is `{ parent, environments, projects }` — the base this config overrides and the scope it applies to — so you can enforce environment-specific rules (e.g. require a field in the `production` override). Example: Require a config to have a name and a non-empty value. ```js if (!config.name || Object.keys(config.value || {}).length === 0) { throw new Error("Configs must have a name and a value."); } ``` ### validateConfigRevision Called before every Config publish, including manual publishes, direct REST updates, automatic publishing after approval, and scheduled publishing. A thrown error blocks the publish and leaves the draft editable. Skipping approval does not skip the hook. Receives: - `config` — the config's published content: `key`, `name`, `project`, staged `value` (a parsed JSON object), `schema`, and lineage (`parent`/`extends`, plus a `lineage` object with `ancestors`, `descendants`, `hasParent`, `hasChildren`, `isRoot`, `isLeaf`). Because a config-scoped hook also runs for descendants, `config.isHookTarget` is `true` only when this is the exact config the hook is pinned to (and `false` for a descendant it inherited); `config.hookTargetKey` names that pinned config (`null` for project/global hooks). Use these to enforce a rule only on the target config, or on the whole family. When the config is an [environment/project override](/features/configs#environment-overrides), `config.scopedConfig` is `{ parent, environments, projects }` (its base and the scope it applies to) — absent otherwise — so you can gate environment-specific rules. - `revision` — the revision being published (when publishing a reviewed draft): `version`, `status`, `comment`, `authorId`, `contributors`, and `reviews`. Each review is `{ userId, decision, comment, stale, dateCreated }` where `decision` is `"approve"` / `"request-changes"` / `"comment"` and `userId` is the reviewer's user ID (or the API key ID for a service account). `revision` is absent on direct (non-draft) writes. Example: Block publishing a value that exceeds a limit. `config.value` is already a parsed object. ```js if ((config.value?.maxItems ?? 0) > 100) { throw new Error("maxItems cannot exceed 100."); } ``` Example: Enforce a rule only on the config the hook is pinned to, not on the descendants it also runs for. ```js if (config.isHookTarget && !config.value?.owner) { throw new Error("The base config must set an owner."); } ``` Example: Enforce an approval policy — require a service-account approval plus at least one human (mirrors the feature-flag `validateFeatureRevision` gate). Disable **Incremental Changes Only** for this hook so it runs on every publish. ```js if (revision) { const approvals = (revision.reviews || []).filter( (r) => r.decision === "approve" && !r.stale ); if (!approvals.some((r) => r.userId === "key_abc123")) { throw new Error("Publishing requires approval from the release-bot service account."); } if (!approvals.some((r) => r.userId !== "key_abc123")) { throw new Error("Publishing requires at least one human approval."); } } ``` Example: Require an external check to pass before publish. ```js const res = await fetch("https://example.com/validate-config", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ key: config.key, value: config.value }), }); if (!(await res.json()).isValid) { throw new Error("External config validation failed."); } ``` ### validateExperiment Called whenever an experiment is about to be created or updated. Receives the full experiment object, including any Custom Field values under `experiment.customFields`. It runs on create, edit, start, and stop, so a thrown error blocks that action. :::note Not every field is populated at creation. An experiment created in the UI starts as a draft, and most of the work — assigning metrics, choosing a hash attribute, configuring variations — happens afterwards, before it is started. The hook still runs on creation, so a rule that requires one of those later fields would block the draft from ever being created. Experiments created through the import flow or the API often do have those fields available up front. If a rule should only apply once an experiment is ready to start, skip drafts early: ```js // Skip draft experiments; validate only when starting or later. if (experiment.status === "draft") return; if (!experiment.hypothesis) { throw new Error("Add a hypothesis before starting this experiment."); } ``` ::: Example: Require at least one experiment surface in a custom field, and require a linked ticket when the "checkout" surface is selected. This validates one multi-select Custom Field against another. ```js const surfaces = experiment.customFields?.experimentSurfaces || []; if (surfaces.length === 0) { throw new Error("Select at least one experiment surface."); } if (surfaces.includes("checkout") && !experiment.customFields?.jiraTicket) { throw new Error("Checkout experiments must reference a Jira ticket."); } ``` Example: Check a Custom Field against an external service. ```js const response = await fetch("https://example.com/validate-experiment", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ team: experiment.customFields?.team }), }); const result = await response.json(); if (!result.isValid) { throw new Error(result.message || "Experiment validation failed."); } ``` Example: Warn without blocking when an experiment has no hypothesis. ```js if (!experiment.hypothesis && experiment.status !== "draft") { addWarning("Consider adding a hypothesis before starting this experiment."); } ``` --- ## Feature Evaluation Diagnostics # Feature Evaluation Diagnostics Feature Evaluation Diagnostics let you inspect recent feature evaluations by querying SDK evaluation events stored in your data warehouse. Use this to verify that any rule (targeting conditions, rollouts, or experiments) is behaving as expected in production. ## Setup To use Feature Diagnostics, send feature evaluation events to your data warehouse. The only required columns are `feature_key` and `timestamp`, but you can include any additional metadata useful for diagnostics, such as `unit_id`, `value`, `reason`, etc. Set up your SDK to send feature evaluation events to your data warehouse using the built-in `onFeatureUsage` callback, as shown in the example snippet below: ```javascript const gb = new GrowthBookClient({ onFeatureUsage: (featureKey, result) => { // Example using Segment analytics.track("Feature Evaluated", { featureKey: featureKey, value: result.value, // Add any additional metadata you want to track reason: result.source, ruleId: result.ruleId, }); } }); ``` Once feature evaluation data is in your data warehouse, set up the feature usage query in GrowthBook. Navigate to your data source, scroll down to **Feature Usage Query**, and add the SQL query for your feature evaluation events. ![Feature Usage Query configuration in data source settings](/images/features/feature-diagnostics/datasource-usage-query-block.png) ## Feature Diagnostics Tab View diagnostic events for your feature flag within the **Diagnostics** tab on any feature. Select your data source and click **View recent feature evaluations** to see the latest 100 feature evaluation events. ![Diagnostics tab showing recent feature evaluation events](/images/features/feature-diagnostics/diagnostics-tab.png) :::tip Query Results Are Not Cached Re-run the query whenever you navigate away from the feature page. ::: ## Troubleshooting If evaluation events aren't appearing in the **Diagnostics** tab, check the following: ### Verify Your `onFeatureUsage` Callback Is Firing Make sure your SDK is correctly configured with the `onFeatureUsage` callback and that events are being sent to your analytics pipeline. You can add a `console.log` statement inside the callback to verify it's being triggered when features are evaluated. ### Check Your Data Pipeline Ensure that events from your analytics tool (e.g., Segment) are flowing correctly into your data warehouse tables. There may be a delay between when events are sent and when they appear in your warehouse. ### Optimize for Large Tables If you have a high volume of feature evaluations, query performance can degrade over time. For optimal performance: - **Partition your table by timestamp** — This allows queries to scan only recent data instead of the entire table. - **Cluster by `feature_key`** — This is especially important for BigQuery users, as it helps queries filter efficiently when looking up evaluations for a specific feature. --- ## Environments import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; # Environments GrowthBook comes with one environment by default (**production**), but you can add as many as you need on the [Environments page](https://app.growthbook.io/environments) located within the **SDK Configuration** menu. :::note Plan availability Free organizations can use the built-in environments (`production`, `dev`, `staging`, and `test`). Creating custom environments beyond these requires a Pro or Enterprise plan. Organizations created before this limit was introduced are unaffected. ::: Feature flags can be enabled and disabled on a per-environment basis. You can also set the default feature state for new environments and scope environments to specific projects. ![A list of environments](/images/features/feature-environments-2.png) When a feature is disabled for an environment, GrowthBook excludes the feature from the API response. The feature evaluates to `null` and ignores all targeting and rules. :::warning When a Feature is Off A feature can be enabled for an environment but still act as "off" if its value is `false`, `null`, `0`, or an empty string. In these cases, even though the feature is enabled, it evaluates as "off". ::: ## Environments and SDKs Each SDK Connection is tied to a specific environment and has its own unique SDK key. When the SDK fetches its configuration, it receives **only the features and rules for that environment**, keeping your production and development environments separate. To use multiple environments in the same codebase, use environment variables to set a dynamic key (e.g., `GROWTHBOOK_CLIENT_KEY='sdk-abc123'`) and reference the variable in your code. ## Environments vs. Projects Environments and projects are both ways to organize feature flags, but they serve different purposes. **Environments** represent _where_ your code runs — such as development, staging, and production. A single feature flag exists across all environments, but can be enabled, disabled, or configured differently in each one. Environments map to SDK connections: each SDK key is tied to one environment, so your production app only receives production rules. **Projects** represent _what_ you're working on — such as a product area, team, or application. Projects scope which flags, experiments, metrics, and data sources are visible to a given team. Items in GrowthBook can belong to multiple projects, and items with no project assigned are available globally. Projects also let you customize permissions and even statistical settings per team. Manage projects under **Settings → Projects**. Free organizations include one project; creating additional projects requires a Pro or Enterprise plan (organizations created before this limit was introduced are unaffected). In practice, you use both together: a feature flag in the "Mobile App" project might be enabled in the dev environment for testing but disabled in production until launch. | | Environments | Projects | | --------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Purpose** | Control _where_ a flag is active | Organize _who_ works on what | | **Scope** | One flag, different settings per environment | Flags, metrics, data sources, and experiments per team | | **SDK impact** | Each SDK connection maps to one environment | SDK connections can be scoped to a project to reduce payload size.**Note:** features and rules can also target projects to scope _delivery_ (advanced — see below) | | **Permissions** | N/A | Users can have different permission levels per project | | **Example** | `dev`, `staging`, `production` | "Mobile App", "Growth", "API" | Learn more about [how to use projects](/using/growthbook-best-practices#projects). :::note Which axis should I use? For most teams, **environments** are the way to control _where_ a flag is active, and projects are mainly for organization and permissions. Projects can _also_ scope delivery — targeting a feature or rule to specific projects — but that's an advanced pattern for cross-project delivery, covered next. Reach for environments first. ::: ### Delivering a feature to multiple projects A feature's **project** is its primary owner. To deliver the same feature into other projects without duplicating it, open the feature and use **Targeting Projects** to add projects (or target all projects). The feature is then included in the SDK payload for any connection scoped to its primary project _or_ one of its Targeting Projects, and appears when you filter the features list by those projects. **Ownership and permissions stay with the primary project.** Who can edit and publish the feature is always keyed on its primary project alone. Adding or removing a Targeting Project is just a normal edit of the feature — it neither requires nor grants any permission in that project, so a user who can't edit or publish in a project can still deliver a feature into it. **Approvals are the exception.** By default, if a Targeting Project has approval requirements of its own, the feature must satisfy them before publishing — so surfacing a feature into a governed project honors that project's review policy. An org-level setting, **Apply approval requirements from Targeting Projects** (in the Approval Flows section of your organization settings), turns this off so only the primary project's approval requirements apply. Either way, only the **primary project's** reviewers can approve; Targeting Projects can raise the review bar but never change who is allowed to review. Individual **rules** can be scoped further: beneath a rule's environment scope, **Project targeting** limits that rule to specific projects within the feature's delivery set — for example, running a rule only in the projects where it's relevant. A rule left unscoped applies to every project the feature reaches. A rule scoped to a project the feature doesn't deliver to won't take effect, and the rule editor flags it. ## Environment Inheritance When you create an environment you can specify a parent environment to inherit from. The new environment initially copies all feature rules from its parent. After creation, the environments operate independently, meaning changes to feature rules in one environment **won't** affect the other. Think of it like creating a fork or snapshot of the parent environment. In the GrowthBook app, you can only inherit rules from the four default environments: `production`, `dev`, `staging`, or `test`. To inherit from a custom-named environment, use the [API endpoint](/api#tag/environments/operation/postEnvironment). ### Common Use Cases - Adding new environments to an existing setup (e.g., adding "staging" after having "dev" and "production") - Troubleshooting production issues by creating an isolated test environment to safely diagnose problems - Creating separate environments on a per-dev or per-PR basis --- ## Feature Flag Experiments import MaxWidthImage from '@site/src/components/MaxWidthImage'; # Feature Flag Experiments GrowthBook allows you to run experiments using feature flags. This method of running experiments allows any feature to be released as an A/B test. It is ideal for more complex experiments requiring multiple code changes, or for companies that want to have an experimentation development culture, and determining the impact on your metrics of any feature or code change. ## Running Experiments with Feature Flags Feature flag experiments are created with an experiment override rule. This experiment rule will randomly assign variations to your users based on the configurations you select. When a user is placed in an experiment via an experiment override rule, the assignment will be tracked in your data warehouse using the `trackingCallback` defined in the SDK implementation. Here's what an Experiment rule looks like in the GrowthBook UI: ![Experiment rule UI](/images/features/feature-experiment-rules.png) This modal window allows for a great deal of flexibility and customization in how you run your experiments. Let's go through each of the options: ### Experiment Targeting Conditions Experiment rules can be targeted at specific user or client attributes. Only users who match the targeting condition will be included in experiment. You can add multiple targeting conditions per rule, and you can add multiple rules per feature, this gives you great flexibility in targeting and customizing your experiment to specific audiences. You can read more about [targeting](/features/targeting). By default, all users will be included. ### Tracking Key The tracking key used to identify this experiment in the SDK. It is used, along with the user hashing attribute in the persistent hashing algorithm to ensure the users always get randomized into the same treatment group. It is also what is passed to the tracking callback to be tracked in your data warehouse. By default, the tracking key is the same as the feature name, but can be any string. We used to use the tracking key as part of the hashing value, but we now instead use a random seed that is generated for each experiment. ### Assign Variations Based on Attribute The value you select here will be hashed along with experiment seed (set randomly for each experiment but fixed for each phase of the experiment) to determine which variation the user will be assigned. Only attributes marked as **identifiers** can be used here. In the vast majority of cases, you want to split traffic based on either a logged-in user id or some sort of anonymous identifier like a device id or session cookie. As long as the user has the same value for this attribute, they will always get assigned the same variation. The values available here are defined in the GrowthBook UI under the _SDK Configuration → Attributes_ section. Like all attributes in GrowthBook, the users value for this attribute must be defined in the SDK implementation. You can read more about targeting attributes [here](/features/targeting). ### Exposure, Variations, and Weights Here you can choose the overall traffic you want to see the experiment as well as any custom split percentages. If you assign to less than 100% of the users, the remaining users will skip the rule and fall through to the next matching one (or the default value) instead. You may want to run an experiment at a split percentages that weights the control group in order to de-risk the new feature (say 90% control, 10% new treatment). It is best practice in such cases to keep the splits the same, and adjust the overall exposure (ie, 20% overall exposure, 50/50 split for the variations). Multiple variations can be added to an experiment from this section as well, as long as the feature is not a boolean. The bar at the bottom shows the traffic allocation for this experiment. Each user will have the experiment's **seed** and **hashing attribute** hashed together to determine which variation they will be assigned. The algorithm is deterministic, and always returns the same value (a number from 0 to 1) as long as the inputs are the same. Changing the split percentages mid-experiment risks having a user switch variations, and could cause multiple exposure warnings. Changing the overall exposure percentage is completely safe. ### Namespaces If you have multiple experiments that may conflict with each other (e.g. background color and text color), you can use **namespaces** to make the conflicting experiments mutually exclusive. Users are randomly assigned a value from 0 to 1 for each namespace. Each experiment in a namespace has a range of values that it includes. Users are only part of an experiment if their value falls within the experiment's range. As long as two experiment ranges do not overlap, users will only ever be in at most one of them. ![Namespaces](/images/namespaces.png) Before you can use namespaces, you must configure them under _Experimentation → Namespaces_. It's essential to ensure that all experiments within a given namespace use the same hash attribute (assignment attribute). ## Experiment Results Once you have the experiment rule saved and published the feature will start to apply these settings and randomize your users into the experiment. The results will flow into your data warehouse, as defined by your `trackingCallback`. There is a link to view experiment results on the bottom of each experiment override rule. Read more about [experiment results](/app/experiment-results). --- ## Feature Flags Feature flags let you control your application's behavior from GrowthBook, without deploying new code. Target specific users, gradually roll out changes, or run A/B tests on the client or the server. Feature flags decouple deploys from releases. Release code more frequently and with less risk by launching to a subset of users, ramping up gradually, or turning any change into an experiment. ![Features Overview](/images/features/features-overview.png) ## Getting Started To use feature flags, set up an [SDK Connection](/lib/quickstart), then explore the docs below. ### Learn the Basics - [Fundamentals](/features/basics) — Keys, feature types, default values, and publishing - [Environments](/features/environments) — Control which features are enabled per environment (dev, staging, production) - [Constants](/features/constants) — Reusable named values you reference from flag values - [Configs](/features/configs) — Typed, validated, composable object shapes you can back JSON flags with (Enterprise) ### Set Up Targeting and Rules - [Rules](/features/rules) — How rules work, choosing a rule type, and configuring forced values, rollouts, experiments, and scheduling - [Targeting Conditions](/features/targeting) — Define who sees a feature using attributes, conditions, and saved groups - [Safe Rollouts](/features/safe-rollouts) — Release features with automatic guardrail monitoring - [Prerequisites](/features/prerequisites) — Gate features based on the state of other features ### Manage Flags at Scale - [Publishing & Approval Flows](/features/publishing-and-approval-flows) — Drafts, revisions, merge conflicts, and requiring approvals for features and saved groups - [Stale Feature Detection](/features/stale-detection) — Identify flags that may no longer be in use - [Code References](/features/code-references) — Find where flags are used in your codebase - [Feature Diagnostics](/features/diagnostics) — Inspect feature evaluations in production - [Custom Hooks](/features/custom-hooks) — Extend GrowthBook with custom validation logic (self-hosted Enterprise) --- ## JSON Schema Validation import CommercialFeature from "@site/src/components/CommercialFeature"; import MaxWidthImage from "@site/src/components/MaxWidthImage"; # JSON Schema Validation JSON feature flags are powerful, especially for remote configuration, letting you ship structured data like pricing rules, layout settings, or experiment parameters without a code deploy. That flexibility comes with a risk: a single typo or missing field in a JSON blob can break production. GrowthBook Enterprise lets you attach a schema to a flag so every value is validated against the structure you define. Validation isn't limited to JSON flags. You can also add a schema to **string** and **number** flags — for example, to restrict a string flag to a set of allowed values or keep a number flag within a min/max range. (Boolean flags can't have a schema.) ![JSON Schema Validation](/images/features/json-schema/validation-banner.png) ## Why Validate? When you're using GrowthBook for remote evaluation, your JSON feature flag values are consumed directly by your application. There's no compiler or type checker between the flag editor and your users. A missing key, a misspelled property, or the wrong type can cause broken UIs, crashed clients, or silent misbehavior. Adding a schema means: - Invalid values can't be saved and mistakes are caught before publishing. - Editors see the exact shape of the data they need to provide. - Consumers of the flag can trust the structure. ## Enabling Validation Open a JSON, string, or number feature flag, go to its **Validation** tab, and find the **Schema Validation** section. Click **Enable** to open the schema editor. You can switch validation on or off at any time, and edit the schema later from the same section. ![JSON Schema Editor](/images/features/json-schema/validation-form.png) ## Two Ways to Define a Schema GrowthBook supports two modes for defining your schema. ### Simple Schema Builder For most configuration flags, a full JSON Schema is more than you need. The Simple Schema Builder is a UI-driven editor for common object shapes, so you don't have to write any schema by hand. Choose a root type: - **Object** - a fixed set of named properties - **Array of Objects** - a list of objects, all matching the same shape - **Primitive Value** - a single string, number, or boolean - **Array of Primitive Values** - a list of primitives (For **string** and **number** flags the value is always a single primitive, so the builder skips this choice and goes straight to configuring that value.) For each property you add, you can configure: - **Property Key** and **Description** - **Type** - Text String, Integer, Float, or Boolean - **Required** (for object properties) - **Restrict to Specific Values** - limit the property to an enum of allowed values - **Min / Max** - bound the length of a string or the range of a number ### Raw JSON Schema When you need more expressiveness than the Simple Builder offers - nested objects, conditional schemas, regex patterns, references, and so on - switch to raw JSON Schema mode and paste in a full [JSON Schema](https://json-schema.org/) document. Anything a standard JSON Schema validator accepts will work here. For string and number flags, the schema's top-level `type` must match the flag — `string` for string flags, `number` or `integer` for number flags. ## The Editing Experience When validation is enabled with a Simple Schema, GrowthBook replaces the raw JSON textarea with a generated form. Instead of editing a JSON blob, editors get a dedicated input for each property: - **Objects** render as a form with one field per property. - **Arrays of objects** render as a table - add and remove rows with buttons. - **Primitives** render as the appropriate input type (text, number, checkbox). - **Enums** render as a dropdown or multi-select with the allowed values. ![Simplified editing UI for simple schemas](/images/features/json-schema/value-editor.png) This form-based editor appears everywhere you set the flag's value: the default value, forced-value rules, rollout rules, and experiment variations. If you ever want to bypass the form, click **Edit as JSON** to switch back to the raw editor. For raw JSON Schemas that are too complex for the form view, editors continue to use the JSON editor, but values are still validated against the schema on save. ## Validation Errors When you try to save a value that doesn't match the schema, GrowthBook shows an inline error with the specific reasons it failed - which property is missing, which value is out of range, which field has the wrong type. Fix the issues and save again. Invalid values can't be published. ## Related - [Feature Flag Basics](/features/basics) — overview of the JSON flag type - [Configs](/features/configs) — reuse one typed, validated schema (and value) across many JSON flags instead of defining a schema per flag (Enterprise) - [Custom Hooks](/features/custom-hooks) — when you need more validation control than a schema provides, run custom server-side validation logic (self-hosted Enterprise) - [Publishing & Approval Flows](/features/publishing-and-approval-flows) — combine schema validation with required approvals for an extra layer of safety --- ## Prerequisite Features import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; Prerequisite features gate other features, [rules](/features/rules), and experiments based on the state of a prerequisite feature. Common use cases: - Grouping multiple related features under a single release feature (e.g., `release-2.8`) and toggling them all at once - Creating a hierarchy of dependent features and safely enabling them in the correct order, per environment - Only enabling a set of features if the user would be bucketed into "variant 1" of an experiment There are two types of prerequisite features in GrowthBook: **Top-level Prerequisites** (feature gating) and **Inline Prerequisite Targeting** (rule-level and experiment-level gating). ## Top-Level Prerequisites Top-level prerequisites are defined per dependent feature. They function similarly to a feature's kill switches: if the prerequisite feature is not serving `true` then the dependent feature will not be enabled. ![Top-level prerequisites](/images/features/feature-top-level-prerequisites.png) Add one or more top-level prerequisites by clicking **Add Prerequisite Feature** on the feature page and selecting the feature you want to use as a prerequisite. For a feature to be eligible as a top-level prerequisite, it must be a boolean (true/false) and in the same project as the dependent feature. ### Prerequisite States and Values A summary of the prerequisite state and value appears while adding or editing a prerequisite, as well as on the dependent feature page. - **Deterministic** states (**live** and **not live**) are applied to your features within GrowthBook. - **Live** prerequisites which are serving `true` will allow their dependent features to be enabled - **Live** prerequisites which are serving `false` will still block their dependent features (`false` ≠ `true`) - **Not live** prerequisites will always block their dependent features (they evaluate to `null`, and `null` ≠ `true`) :::info Prerequisites with deterministic states work "out of the box" regardless of SDK version support. Any feature that is always "not live" will not be seen by the SDK. Any feature that is always "live" will no longer reference its prerequisites in the SDK. This means that no SDK-level evaluation of prerequisites is needed (these prerequisites work irrespective of SDK version support). ::: - **Schrödinger** state means that the prerequisite's state cannot be determined in advance. It may depend on user attributes or other non-deterministic factors. (This is homage to the physicist Erwin Schrödinger who proposed a thought experiment involving a cat in a box that is both alive and dead at the same time.) :::info Prerequisites with a Schrödinger state must be evaluated at runtime in the SDK. Prerequisite evaluation is supported in the following SDK versions: - JavaScript: `0.34.0+` - React: `0.24.0+` - Python: `1.1.0+` - C#: `1.1.0+` - Golang: `0.2.0+` - Java: `0.9.7+` - Kotlin: `1.1.60+` - Swift: `1.0.56+` - Ruby: `1.3.0+` ::: ## Inline Prerequisite Targeting Inline prerequisite targeting provides finer-grained control than top-level prerequisites. 1. Inline prerequisites can be applied at a feature's [rule](/features/rules) level, and can be environment-specific. This comes with the added benefits of feature draft releases and approvals. 2. Inline prerequisites can be applied to individual experiments which may not be linked to a specific feature (such as visual experiments). 3. Inline prerequisite targeting is not limited to boolean features. You can specify any evaluation condition you'd like (ex: prerequisite value is: greater than 3, in a list of allowed values, or matches a regex pattern). You can even do advanced targeting with JSON. To create an inline prerequisite within a feature, add prerequisite targeting to an existing rule or create a new rule with prerequisite targeting. You can specify one or more prerequisite features within the same project and give each a custom evaluation condition. A similar flow exists while editing the targeting rules of an experiment. ![Inline prerequisite targeting](/images/features/feature-inline-prerequisite-targeting-2.png) ### Inline Prerequisite States and Values The same **deterministic** and **Schrödinger** states apply to inline prerequisites as they do to top-level prerequisites ([see above](#prerequisite-states-and-values)). Below is a summary of how they apply to inline prerequisites: - **Deterministic** states (**live** and **not live**) are calculated using your evaluation condition, which is **_not_** limited to `is true`. As before, no run-time evaluation of prerequisites is required in the SDK. - **Live** prerequisites which pass the evaluation condition will allow their dependent rules or experiments to be enabled. - **Live** prerequisites which fail the evaluation condition will still block their dependent rules or experiments. - **Not live** prerequisites will generally block the dependent rule or experiment, unless the evaluation condition specifically checks for this (e.g. `is not live`) - **Schrödinger** state prerequisites must be evaluated at runtime in the SDK, and thus a compatible SDK version is required. ## Limitations GrowthBook enforces the following limitations when configuring prerequisites: 1. Prerequisite features must be in the same project as the dependent feature or experiment. 2. You cannot select a prerequisite that would lead to a circular dependency. 3. If you don't have an SDK which supports prerequisite evaluation, then you cannot select a prerequisite that is in a Schrödinger state. 4. Once a feature has been used as a prerequisite for other features or experiments, you are blocked from deleting, archiving, or changing its projects. To perform these actions, you must first remove the feature from all dependent features and experiments. You can see a list of dependencies on the feature page: ![Prerequisite dependents](/images/features/feature-prereq-dependents.png) --- ## Publishing & Approval Flows import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; # Publishing & Approval Flows When you make changes to a feature flag or [saved group](/features/targeting#saved-groups) in GrowthBook, those changes go through a publishing workflow before they reach your users. This page covers the full lifecycle: drafts, publishing, revisions, merge conflicts, and optional approval flows. The same revision and approval model applies to both feature flags and saved groups. The sections below describe the flow for feature flags first; the [Saved Groups](#saved-groups) section at the end calls out where the saved group experience differs. ## Drafts and Publishing When you change a feature's definition (e.g., the default value or rules), GrowthBook automatically creates a new draft revision. This draft is unpublished and only visible within the GrowthBook UI, not to your users. Continue adding changes to the draft, and when you're ready, publish them all at once with an optional commit message. ![Draft Modal](/images/features/feature-publishing-2.png) ## Revisions Once published, a revision becomes locked and can no longer be changed. To undo changes and revert to a previous state, use the **Revision** dropdown to select the version you want to revert to, then click **Revert**. You can review the changes that will be applied before submitting. ![Feature Revisions](/images/features/revision-dropdown-2.png) ## Merge Conflicts Your draft can diverge from the live version of a feature if someone else publishes changes while you're still editing. GrowthBook can merge some changes automatically (e.g., if you change a rule in "dev" and someone else changes a rule in "production"). For changes that can't be resolved automatically, you must fix the conflicts before publishing your draft. The conflict resolution process works similarly to version control systems like Git. For each conflict, GrowthBook shows a diff and lets you choose how to proceed. In the example below, the Default Value was set to `foo` when you first created your draft. In your draft, you changed the value to `bar`. At the same time, someone else published a new version, changing the value to `hi!!`. So, now you have to choose which change you want to keep. ![Resolve Conflicts](/images/features/resolve-conflicts-2.png) ## Approval Flows With Approval flows, you can require approval before publishing any change to an existing feature flag. Approval flows help reduce errors by making sure changes to features have been viewed and approved by someone else in your organization. ### Settings Enable Approval Flows for your organization on the settings page. Select the environments that require approvals, or leave the field blank to require approvals on all environments. To force-reset the review when a change is made after approval, toggle on **Reset review on changes**. ![Approval flow organization settings](/images/features/approval-flows/org-settings.png) ### Requesting a Review When Approval Flows are enabled, you must request a review before publishing changes. Add a detailed comment describing the changes so reviewers can understand the intent of your request. ![Requesting a review for feature changes](/images/features/approval-flows/request-approval-to-publish.png) Requested reviews appear in the **Drafts** tab on the Features overview page with a **Pending Review** status. Sort by date updated to see your most recent request at the top. ![Pending review list in the Drafts tab](/images/features/approval-flows/drafts-pending-review-list.png) ### Reviewing A reviewer needs the **Review** permission for Feature Flags. The person who created the draft cannot approve it. If **Require approval from a non-editor** is enabled, anyone who contributed to the draft is also prevented from approving it. See [Permissions](/account/user-permissions) for details. Click the feature, then click **Review and Approve** to open the review modal. ![Review and Approve button on a feature](/images/features/approval-flows/review-and-approve-button.png) From here, you can see the diff between the currently published changes as well as the comments. ![Diff view showing changes between published and draft versions](/images/features/approval-flows/review-draft-changes-diff-2.png) Click **Next** to write your review in the comment box and select a status: - **Comment** — Add a note without formally reviewing the changes - **Request Changes** — Flag issues that need to be addressed before approval - **Approve** — Confirm that everything looks correct ![Review status options: Comment, Request Changes, Approve](/images/features/approval-flows/review-status-options.png) Once the changes are **Approved**, they can be published. ![Approved draft ready to publish](/images/features/approval-flows/approved-ready-to-publish.png) ### Publishing with Approvals After approval, anyone with **Publish** access to the affected environments can publish the draft. A user with **Bypass draft approvals** can publish without an approval. ![Publishing approved changes with admin bypass option](/images/features/approval-flows/publish-with-admin-bypass.png) Once published, a green "Live" badge confirms the version is published and active. ![Green badge indicating the published version is live](/images/features/approval-flows/revision-live-badge.png) ### Visualize Changes with Compare Revisions Get a detailed diff of the changes between published and draft versions by clicking the **Compare Revisions** button. ![Compare Revisions button](/images/features/approval-flows/compare-revisions-diff.png) ## Saved Groups Saved groups follow the same draft → review → publish lifecycle as feature flags. Every change you make to a saved group — editing values or conditions, updating metadata, or archiving it — is tracked as a revision so you can review, compare, and roll back changes over time. ### Saved Group Revisions When you edit a saved group, GrowthBook creates a draft revision. Drafts are visible to everyone with access to the saved group, but they don't affect SDK evaluations until they're published — so you can stage and review your changes before they reach your users. From the saved group detail page you can: - **Switch revisions** — Use the revision dropdown to view the live revision or any open draft. The dropdown shows the author, last update date, and review status for each revision. - **Create a new draft** — Click **New Draft** from the live revision to start a new set of changes. - **Compare revisions** — Click **Compare revisions** to see a side-by-side diff of any two revisions. - **Revert to a previous revision** — Open an older merged revision and click **Revert to Previous** to apply its state as a new draft. - **Discard a draft** — Authors can discard their own draft if they no longer want to publish it. - **Fix merge conflicts** — If another user publishes changes while your draft is open, click **Fix conflicts** to resolve them before publishing. The conflict resolution flow works the same way it does for features. ### Saved Group Approval Flows You can require approvals for saved group changes independently from feature flag approvals. #### Settings Enable saved group approvals from **Settings → General → Approval Flows**. The Saved Groups section has its own toggle, separate from the Features section: - **Require approval to modify Saved Groups** — When enabled, every change to a saved group must be reviewed and approved by someone else before it goes live. - **Require approval for** — Approval is always required for changes to values and conditions. Use the **Metadata changes** checkbox to also require approval for edits to the description, owner, project, or tags. - **Reset review on changes** — If a draft is modified after it has been approved, the approval is revoked and a new review is required before publishing. - **Require approval from a non-editor** — Anyone who edited the draft is blocked from approving it, ensuring a separate reviewer must approve before publishing. #### Requesting and Reviewing When approvals are enabled, **Review & Publish** becomes **Request Approval to Publish**. The request appears in the Saved Group's revision list with a **Pending Review** status. A user with **Review** access can open the request, inspect the diff, comment, request changes, or approve it. The draft's creator cannot approve it. When **Require approval from a non-editor** is enabled, other contributors cannot approve it either. After approval, a user with **Publish** access in every Project assigned to the Saved Group can publish it. **Bypass draft approvals** allows the same user to publish without an approval. #### Archiving and Unarchiving Archiving and unarchiving a Saved Group always creates a revision. When approval flows are enabled, that revision must be approved before it takes effect. Archiving requires **Archive & delete** access; unarchiving requires **Publish** access. A Saved Group with active references from Feature Flags, experiments, or other Saved Groups cannot be archived until those references are removed. --- ## Ramp Schedules import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from "@site/src/components/CommercialFeature"; A Ramp Schedule rolls a feature out in stages instead of switching it on for everyone at once. You define the stages up front, and GrowthBook moves through them on a schedule you set. A typical schedule might show the feature to 1% of users, then 5%, then 25%, then 100%. Between stages, the rollout can wait a set amount of time or pause for a teammate's approval, so you widen exposure gradually and can stop early if something looks off. ## How ramp schedules work A Ramp Schedule attaches to a single **Targeting rule** (a rule that serves a value to matching users, with optional targeting conditions). It is a release plan on top of that rule, not a separate rule type. You add one from the rule editor by choosing **Ramp-up** as the release plan. The schedule drives the rule through an ordered list of **steps**. Each step only sets the fields you want to change; everything else carries over from the previous step. A step can change: - **Rollout %**: the percentage of matching users who get the rule's value. - **Targeting**: the condition, saved groups, and prerequisites. - **Environments**: which environments the rule applies to. - **Value**: the value the rule serves. Each step also has an **Action** that controls when it advances: - **Hold for**: apply the step, then wait a set amount of time (for example, 12 hours) before advancing. - **Hold for approval**: apply the step, then wait for a teammate to approve before advancing. You can also add **+ Approval** to a timed step so it waits on both. Approval is always the final gate, so any time or sample-size conditions clear first. - **Hold for min. sample**: wait until a set number of users have been exposed. This applies only to [monitored steps](#monitored-steps-and-guardrails). When a step's Action clears, GrowthBook applies the step's changes to the rule and starts the next step. Each advance publishes a new feature revision automatically, so your SDKs pick up the change through the normal feature payload. If a schedule falls behind (for example, background processing was interrupted past several steps' hold times — pausing doesn't cause this, since pause shifts the timers), the overdue steps are caught up in a **single** advance: one revision publish (labeled with the folded range, such as "Ramp steps 4–12 of 20") and one webhook whose `previousStepIndex` shows the gap — `rampSchedule.actions.step.advanced`, or `rampSchedule.actions.completed` when the fold finishes the ramp. Consumers mirroring ramp progress should diff `currentStepIndex - previousStepIndex` rather than counting events. Approval and monitored steps are never skipped by a catch-up — the schedule always stops at them. The grid always ends with an **end** row, set to 100% by default. That is the state the rule lands in when the schedule finishes; open its menu to attach final rule changes, such as removing targeting. ![Ramp-up step grid with rollout percentages, hold intervals, and per-step actions](/images/features/ramp-schedule-steps.webp) ### Schedule timing - **Start**: choose **Immediately** to begin as soon as the rule is published, or **On date** to delay activation. The rule stays disabled until the start date. - **Duration**: the total length of the ramp. In **Simple View** you set it directly and GrowthBook spaces the steps to fit; in **Advanced View** it shows a computed summary of your steps, such as "~5d + monitored steps". - **Disable on date**: click **+ Disable on date** to set an optional date that turns the rule off whether or not the ramp has finished. Use it for time-boxed rules. ### Lock feature while running Turn on **Lock feature while running** to block publishing other draft changes to the feature while the ramp is actively progressing. This keeps manual edits from competing with the schedule. :::note The lock only applies while the ramp is running. It does not apply when the ramp is paused, completed, or rolled back, and it never blocks the changes the ramp engine makes as it advances. Pause the ramp to make immediate changes. ::: ### Sample by and hashing **Sample by** sets the attribute GrowthBook hashes to assign users to the rollout, so the same user stays in the same bucket as the rollout grows. Expand **Hashing & seed options** to set a custom **Seed** or pick the **Hashing** algorithm version. These work the same as on a standard rollout rule. ## Ramp schedule vs. Safe Rollout A [Safe Rollout](/features/safe-rollouts) is a Ramp Schedule with guardrail monitoring turned on. Both use the same engine. The difference is whether you watch metrics as the ramp progresses. - Use a plain **Ramp Schedule** when you want a release plan that runs on its own: step the rollout up over time, broaden targeting in stages, gate jumps behind approval, or disable the rule on a set date. - Use a **monitored** Ramp Schedule (a Safe Rollout) when you also want GrowthBook to watch guardrail metrics and automatically hold or roll back if the release harms them. You configure monitoring per step, so a single schedule can mix unmonitored and monitored steps. ## Adding a ramp schedule ### 1. Add a Targeting rule When you add a rule to a feature, choose **Targeting rule** as the rule type. ![Rule type picker showing Targeting rule, Experiment, and Bandit, with a callout pointing to Safe Rollouts](/images/features/rule-type-picker.webp) :::tip Looking for Safe Rollouts? Choose a Targeting rule and turn on guardrail monitoring in its ramp-up schedule. The **Show me** shortcut on the rule type screen configures a fully monitored ramp for you. ::: ### 2. Choose the Ramp-up release plan In the rule settings, set the release plan to **Ramp-up** (or **Monitored Ramp-up** to watch guardrail metrics). The ramp editor opens so you can define the steps. ### 3. Configure the steps The step editor opens in one of two modes: - **Simple View**: pick a total **Duration** and GrowthBook spreads a standard set of steps (1%, 5%, 10%, 25%, 50%, then 100%) across it. Good for a quick, standard ramp. - **Advanced View**: edit the step grid directly. Each row sets a **Rollout %** and an **Action**, and you can add per-step approval, monitoring, or rule changes. Switch between modes with the **Edit Ramp-up Steps** and **Simple View** buttons. If your org has saved **Ramp Schedule Templates**, choose one from the **Template** dropdown to prefill the steps, then adjust from there. The dropdown appears only when templates exist. While a ramp is running, the schedule controls the rule's rollout. Pause or end the ramp to make immediate changes. ### 4. Set the start and disable dates Optionally set **Start** to **On date** to delay activation, and use **+ Disable on date** to turn the rule off at a fixed time. ### 5. Publish Publish the revision to arm the schedule. A schedule configured inside a draft stays in the `pending` state until that draft is published. ## Monitored steps and guardrails Mark a step as **monitored** (the shield icon on the step) to run guardrail analysis while it is active. GrowthBook splits enrolled users 50/50 between the new value and the existing value and analyzes your guardrail metrics, using the same engine as a [Safe Rollout](/features/safe-rollouts). On a monitored step, the **Rollout %** is the share of users who get the new value, and an equal-sized control group gets the existing value. Rollout % is capped at 50% on monitored steps: at that point the new value and the control each reach half your users, so no one is left out. :::note A monitored step at **50%** is a full 50/50 split: half your users get the new value and half get the existing value. Set it to **25%** to show the new value to 25% of users, with a 25% control. ::: Turn on **Monitor this release** and configure monitoring once on the schedule: - **Data source** and **Assignment table**: where traffic and metric data come from. - **Guardrail Metrics**: GrowthBook automatically rolls back and disables the rule if any of these show a significant regression. - **Signal Metrics**: GrowthBook pauses at the current step if any of these regress. You resume manually, or it resumes automatically when the metric recovers. - **Refresh results every**: how often GrowthBook re-analyzes your metrics. Leave it blank to use the org default (6 hours). Under **Advanced Settings**, choose what happens when an automated check fails: | Check | UI control | Options | | --------------------- | ------------------------- | ----------------------------------------- | | Sample ratio mismatch | **If SRM detected** | Hold step (default), Roll back, Warn only | | No traffic | **If no traffic** | Hold step (default), Roll back, Warn only | | Multiple exposures | **If multiple exposures** | Hold step (default), Roll back, Warn only | **Hold step** pauses the ramp for review, **Roll back** rewinds the rule to its pre-ramp state and ends the schedule, and **Warn only** flags the issue without stopping the ramp. The no-traffic grace period defaults to 24 hours and is editable next to **If no traffic**. Guardrail regressions always roll back and disable the rule; that behavior is not configurable here. :::note Sticky bucketing is disabled on monitored steps, because the hash ranges shift as the rollout grows. **Hold for min. sample** applies only to monitored steps, since unmonitored steps have no analysis to evaluate. ::: ## Operating a running schedule Once started, a schedule reports its status and progress on the feature's rule. ![A running ramp schedule on a feature rule, showing the current rollout percentage, the served value, the current step, time remaining, and the step timeline](/images/features/ramp-schedule-timeline.webp) | Status | Meaning | | ------------- | --------------------------------------------------------------- | | `pending` | Created but not yet armed. Publish the revision to start. | | `ready` | Armed and waiting for the start date. | | `running` | Actively advancing through steps. | | `paused` | Halted by a user or a `hold` health action. Resume to continue. | | `completed` | All steps applied and the end state set. | | `rolled-back` | Returned to the pre-ramp state. | From the UI or API you can start, pause, resume, manually advance, jump to a specific step, approve a gated step, roll back to the pre-ramp state, or restart a finished schedule. Every transition is recorded in the schedule's event history. ## Templates Use **Save as template** to store a schedule's steps and end state as a **Ramp Schedule Template** and reuse them across features. Apply a template when you create a schedule to inherit its steps, then override any of them as needed. ## Managing via the API Ramp Schedules have a full REST API under `/ramp-schedules`. This example attaches a three-step ramp to an existing rule, with the final step gated on approval: ```bash curl -X POST https://api.growthbook.io/api/v1/ramp-schedules \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Checkout v2 rollout", "featureId": "checkout-v2", "ruleId": "fr_abc123", "steps": [ { "interval": 3600, "actions": [{ "patch": { "coverage": 0.1 } }] }, { "interval": 86400, "actions": [{ "patch": { "coverage": 0.5 } }] }, { "interval": null, "holdConditions": { "requiresApproval": true }, "actions": [{ "patch": { "coverage": 1.0 } }] } ] }' ``` When you pass both `featureId` and `ruleId`, GrowthBook fills in the target on every action, so you only supply the fields you want to change. See the [API reference](/api) for the full set of endpoints and lifecycle actions. ## What's next - [Safe Rollouts](/features/safe-rollouts): a Ramp Schedule with guardrail monitoring. - [Rules](/features/rules): choose the right rule type before adding a schedule. - [Publishing and approval flows](/features/publishing-and-approval-flows): gate rule changes on reviewer approval. --- ## Feature Flag Rules import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; # Rules Every feature has a [default value](/features/basics#default-values) that applies to all users. **Rules** override that default for the users, environments, and projects you choose. ## How Rules Work A feature's rules are an ordered list. For each request, GrowthBook evaluates them top to bottom and the **first matching rule wins**. If no rule matches, the default value is used. ![Feature rules interface](/images/features/feature-override-rules-2.png) Each rule has up to four parts: - **Targeting conditions** define _who_ the rule applies to. Conditions are evaluated against the [attributes](/features/targeting) you pass into the SDK. A rule with no conditions applies to everyone. - **What it does** for matching users — serve a value, run an experiment, or run a bandit. - **Scope** — which [environments](#environments) and, optionally, which [projects](#projects) the rule applies to. - **Release plan** (optional) — a value-serving rule can [ramp up automatically](#ramp-ups-and-safe-rollouts) over time instead of switching on all at once. :::tip Rules can be combined Layer multiple rules on one feature to compose behavior — for example, a forced value for beta testers above a gradual rollout to everyone else. ::: ## Rule Types | | Targeting rule | Experiment | Bandit | | ----------------- | ----------------------------------- | --------------------------------- | ---------------------------------------- | | **Purpose** | Serve a value to matching users | Measure which value performs best | Auto-optimize toward the best value | | **Randomization** | Optional (percentage rollout) | Random split | Random split, reweighted over time | | **Tracking** | No | Yes | Yes | | **Typical use** | Enable for a group; gradual release | A/B test a change | Maximize a metric across several options | Gradual ramp-ups and guardrail-monitored "safe" rollouts are **not** separate rule types — they're [release plans](#ramp-ups-and-safe-rollouts) you add to a targeting rule. ### Targeting Rule (forced value or percentage rollout) The most common rule serves a single value to the users who match its conditions. In the rule editor it appears as two presets of the same rule: - **Forced value** — everyone who matches gets the value. - **Percentage rollout** — a random sample gets the value and the rest fall through to the next rule. A forced value is simply a 100% rollout. ![Forced value rule UI](/images/features/feature-force-rule-2.png) For a percentage rollout, choose a user attribute (e.g., `id` or `company`) to sample on. Users with the same attribute value always get the same experience — choosing `company`, for example, keeps everyone at a company on the same side of the rollout. ![Percentage rollout UI](/images/features/feature-rollout-rule-2.png) Common scenarios: - Turning a feature on for internal employees or beta testers - Overriding a value for a single account that reported a bug - Releasing to 10% of users, then 50%, then 100% To widen a rollout **automatically** instead of editing the percentage by hand, add a [Ramp-up release plan](#ramp-ups-and-safe-rollouts). ### Experiment Experiment rules randomly split users into variations, assign each variation a value, and log every assignment through the SDK's `trackingCallback` so you can measure impact in your data warehouse or analytics tool. ![Experiment rule UI](/images/features/feature-experiment-rules-2.png) You control both the percentage of users included and the split between variations. For example, including 50% of users with a 40/60 split means 20% of all users see variation A, 30% see variation B, and the remaining 50% fall through to the next rule. → [Full experimentation documentation](/features/feature-flag-experiments) ### Bandit A [Bandit](/bandits/overview) rule works like an experiment but **automatically shifts traffic toward the better-performing variation** as results come in, rather than holding a fixed split. Use it to optimize toward a metric across several options instead of measuring one change precisely. :::note **Contextual Bandits**, which choose a variation per user based on their attributes, are in closed beta. ::: ## Ramp-ups and Safe Rollouts A **Ramp Schedule** is a release plan you attach to a value-serving (targeting) rule — not a separate rule type. Choose **Ramp-up** as the rule's release plan to roll out in timed, optionally approval-gated stages that advance on their own (for example, 1% → 5% → 25% → 100%). You widen exposure gradually and can stop early if something looks wrong. This is the modern replacement for manually bumping a rollout percentage. → [Ramp Schedules](/features/ramp-schedules) **Safe Rollout** is the same mechanism with **guardrail monitoring** turned on: GrowthBook runs the ramp as a short A/B test against your guardrail metrics (error rate, latency, conversions, …) and flags regressions so you can roll back. If you've used a standalone "Safe Rollout" rule before, it's now a monitored ramp-up rather than its own rule type. → [Safe Rollouts](/features/safe-rollouts) ## Scoping Rules ### Environments Each rule applies to **all environments or a specific set**, chosen in the rule's environment scope. Because each [SDK connection maps to one environment](/features/environments#environments-and-sdks), this controls where a rule actually takes effect — letting you test a rule in `dev` before enabling it in `production`. ### Projects A rule can optionally be scoped to specific **projects** within the feature's delivery set — an advanced way to run a rule in only some of the projects a multi-project feature reaches. A rule left unscoped applies to all of them; a rule scoped to a project the feature isn't delivered to won't take effect (the editor flags it). → [Delivering a feature to multiple projects](/features/environments#delivering-a-feature-to-multiple-projects) ## Scheduling Rules Any rule can be scheduled to turn on or off at a specific date and time — useful for launches, holidays, or special promotions. ![Feature Scheduling](/images/features/feature-scheduling-2.png) For a multi-stage rollout that advances automatically, use a [Ramp Schedule](#ramp-ups-and-safe-rollouts) instead. ## Targeting Conditions Any rule can include targeting conditions to limit which users it applies to. Conditions you define in GrowthBook are evaluated against the attributes you pass into the SDK. ![Rule conditions UI](/images/features/feature-rule-conditions.png) → [Full targeting reference: attributes, conditions, and saved groups](/features/targeting) ## Testing Rules Test your rules directly in GrowthBook on the **Simulation** page. Adjust user attributes and see in real time which rules match and what value the user would receive. ### Archetypes Archetypes let you save preset user attribute profiles so you can quickly test how rules apply to specific types of users. If you frequently target features to certain groups (e.g., beta testers, enterprise accounts), archetypes let you check the result in one click. They appear at the top of the **Simulation** page. Hover over any value to see debug information. ![Archetypes](/images/features/feature-archetypes-2.png) Create and manage archetypes by navigating to **SDK Configuration** → **Archetypes**. --- ## Safe Rollouts import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from "@site/src/components/CommercialFeature"; Releasing a new feature always carries risk. Even small changes can introduce regressions. Safe Rollouts reduce this risk by releasing to a subset of users while monitoring key metrics for issues. ## Overview A Safe Rollout is a [Ramp Schedule](/features/ramp-schedules) with guardrail monitoring turned on. It releases a feature to a subset of users and automatically monitors guardrail metrics for regressions. Guardrails can be any metrics that matter to your team, such as error rates, latency, or conversions. The rollout runs as a short-term A/B test: - The **Control** receives the existing value. - The **Rollout** receives the new value. GrowthBook analyzes the rollout's impact on your selected guardrail metrics and provides status updates: safe to ship, issues detected, or consider rolling back. ## How to Add a Safe Rollout ### 1. Add a Targeting Rule with Monitoring ![Rule type picker with the Looking for Safe Rollouts callout and Show me shortcut](/images/features/rule-type-picker.webp) When adding a new rule to your feature, select **Targeting rule**, then choose **Monitored Ramp-up** as the release plan. The **Show me** shortcut on the rule type screen configures a fully monitored ramp for you. :::note Safe Rollouts are now configured as a monitored [Ramp Schedule](/features/ramp-schedules) rather than a separate rule type. The monitoring, analysis, and rollback behavior described below is unchanged. ::: ### 2. Configure the Safe Rollout - **Rollout value:** the value users in the Rollout group receive. It is compared against the Control (the rule's existing value). - **Sample by:** the attribute used to assign users to the Control and Rollout groups (for example, `id` or `organization_id`). ### 3. Define Metrics and Monitoring ![Monitored ramp-up editor showing the Monitor this release toggle, data source, assignment table, and guardrail and signal metrics](/images/features/safe-rollout-create.webp) - **Data Source and Assignment Table:** Confirm the data source and assignment table used for exposure tracking. - **Guardrail Metrics:** Select one or more metrics to monitor for regressions. GrowthBook automatically rolls back and disables the rule if any of them regress significantly, so there is no separate auto-rollback toggle. - **Signal Metrics:** Optional metrics that pause the rollout at the current step if they regress, without rolling it back. It resumes when they recover. Under **Advanced Settings**, you can choose how GrowthBook responds to other issues, such as a sample ratio mismatch, no traffic, or multiple exposures: hold the ramp, roll back, or warn only. ### 4. Ramp Up Schedule A Safe Rollout ramps traffic up through the steps of its underlying [Ramp Schedule](/features/ramp-schedules). By default, a monitored ramp steps the rollout through: **1% → 5% → 10% → 25% → 50%** Each monitored step splits its traffic 50/50 between the Rollout and the Control, so monitored steps top out at 50%. The early steps each hold for a short slice of the **Duration**, then the rollout holds at 50% for the rest of the window before releasing to 100% when the schedule completes. You can change the pace with **Duration**, or edit the steps directly in **Advanced View**. ### 5. Monitor the Rollout ![Feature rule card showing a Safe Rollout with a Guardrails Failing badge and a Revert Now button](/images/features/safe-rollout-rollback-now.webp) Once published, GrowthBook begins monitoring the guardrail metrics. The rule shows a status badge that updates automatically as results come in: | Status | Meaning | | ------------------ | ------------------------------------------------------------------------------------------------------------- | | X days left | The rollout is running. Monitoring continues until the duration completes. | | Unhealthy | A health check failed, such as a sample ratio mismatch. Traffic may be imbalanced; check your implementation. | | Guardrails Failing | A guardrail metric is regressing. The rule shows a **Revert Now** button so you can roll back. | | Ready to ship | The duration finished with no failing guardrails. Safe to release to 100%. | | No Data | No traffic detected. Check your setup. | | Reverted | The rollout was rolled back to the control value. | | Released | The rollout was shipped to all users. | :::info Guardrails are analyzed for failure using frequentist sequential testing, allowing you to roll back as soon as statistical significance is reached without fear of false positives. Your safe rollout is automatically monitored for implementation errors with sample ratio mismatch and multiple exposures checks. ::: ### 6. Take Action At the end of the monitoring period (or sooner if issues arise), you can: - Release the feature to 100% of users (if not already there). - Revert the feature to the control value. - Continue to monitor. If a guardrail metric regresses, GrowthBook automatically rolls back and disables the rule to protect your users. ## Understanding the Time Series Graph The safe rollout monitoring interface includes a time series graph showing how your guardrail metrics perform over time. The graph displays two key values: ### Metric Boundary The Metric Boundary is the statistical boundary for whether a safe rollout is failing. When it crosses zero, we have enough statistical certainty that the safe rollout is harming this metric. Technically, it is the [lower/upper] bound of the absolute change confidence interval between the baseline and the safe rollout groups. ### Threshold The threshold for when a metric is considered failing. It's always set to zero — as soon as there's statistical certainty that a metric is being harmed at all (even by very small amounts), the safe rollout is marked as failing. ## FAQs ### How Does the Traffic Ramp Up Work? A Safe Rollout ramps traffic through the steps of its [Ramp Schedule](/features/ramp-schedules). By default it steps through 1%, 5%, 10%, 25%, and 50% before releasing to 100%, keeping the initial blast radius small and widening as monitoring stays healthy. You set the pace with the rollout's **Duration** and can customize the steps in **Advanced View**. ### What Happens if a Guardrail Metric Degrades? If a guardrail metric crosses the significance threshold for a regression, GrowthBook rolls back and disables the rule automatically to prevent further harm, and the rollout's status shows the failing guardrail. For the other automated checks (sample ratio mismatch, no traffic, and multiple exposures), you choose the response under **Advanced Settings**: hold the ramp, roll back, or warn only. ### What Should I Use as My Guardrail Metrics? Choose metrics that represent key system health or business outcomes. This might include error rates, latency, or conversion rates—whatever signals that the change is working as intended and not causing harm. Choosing too many guardrail metrics increases the chance of false positives. Aim for a focused set of critical metrics. ### How Is This Different From an Experiment? Safe Rollouts use the same analysis engine as GrowthBook experiments but are designed for operational decision-making, not learning. The primary goal is to ensure a safe release, not to measure long-term impact. ### What Happens if There Isn't Enough Data? Safe Rollouts are built to help you ship confidently and reduce the chance of negative regressions. If any guardrail fails, roll back the change. If results are still inconclusive after the configured duration, ship — there's no clear evidence that the feature is harmful. Safe Rollouts bias towards action. If you're more uncertain about a feature and want to learn about its impact, run a regular [Experiment](/features/rules#experiment) instead. --- ## Saved Group Revisions API import CommercialFeature from '@site/src/components/CommercialFeature'; # Saved Group Revisions API The saved group revisions REST API is the programmatic counterpart to the UI workflow described in [Publishing & Approval Flows](/features/publishing-and-approval-flows#saved-groups). Every change goes through the same draft → review → publish lifecycle, so a draft you start via REST can be reviewed in the UI (and vice versa). If you just need to read or replace a saved group atomically — and your org does not require approvals — keep using [`POST /saved-groups/{id}`](/api/#operation/updateSavedGroup) with the `bypassApproval: true` flag. The endpoints below are for callers who need to stage changes, run approvals, or coordinate with the UI revision flow. The full endpoint reference lives in the [REST API docs](/api/#tag/saved-group-revisions). This page covers the lifecycle and the request shapes you'll typically reach for. ## Drafts and Publishing When you edit a saved group via the revision endpoints, GrowthBook creates a draft revision. Drafts hold proposed changes against a snapshot of the saved group at the moment the draft was opened — they don't affect SDK evaluations until they're published. Open a new draft explicitly: ```bash curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"title": "Add Q2 beta cohort"}' ``` The response contains the new revision and its integer `version`, which you pass to subsequent edit calls: ```json { "revision": { "id": "rev_...", "version": 4, "status": "draft", "baseSavedGroup": { "...": "snapshot taken when the draft opened" }, "proposedSavedGroup": { "...": "snapshot + proposed changes applied" }, "proposedChanges": [] } } ``` Stage a change on the draft. For list saved groups, prefer the incremental endpoints — they're idempotent and stack on top of the current draft state, so multiple add/remove calls accumulate: ```bash curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/items/add' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"items": ["user_42", "user_77"]}' ``` For an atomic full replacement, use `PUT .../values`. For condition saved groups, use `PUT .../condition`. Metadata edits (name, owner, description, projects) go through `PUT .../metadata`. Archive or unarchive with `PUT .../archive`. Publish to apply the draft to the live saved group: ```bash curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/publish' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{}' ``` After publish, the revision status becomes `merged` and the proposed changes are applied to the live saved group. ### Auto-creating a draft on edit Every field-edit endpoint (`PUT .../metadata`, `PUT .../condition`, `PUT .../values`, `PUT .../archive`, `POST .../items/add`, `POST .../items/remove`) accepts the literal `"new"` in place of a version number. This opens a fresh draft, applies the edit, and returns the revision in one round trip. Pass `revisionTitle` / `revisionComment` to label the auto-created draft: ```bash curl -X PUT 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/new/metadata' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "revisionTitle": "Rename to internal-beta", "name": "internal-beta", "description": "Cohort moved to the internal-only beta program" }' ``` `revisionTitle` and `revisionComment` are ignored when editing an existing draft. ## Revisions A saved group has one live revision and zero or more open draft revisions. Each revision has a status: | Status | Meaning | | ------------------- | ------------------------------------------------------------------------- | | `draft` | Open, editable. Authors can keep staging changes. | | `pending-review` | Submitted for review. Reviewers can approve, comment, or request changes. | | `changes-requested` | A reviewer asked for changes. Edit the draft and re-request review. | | `approved` | Approved and ready to publish. | | `merged` | Published. Terminal. | | `discarded` | Abandoned. Terminal. | Use these endpoints to read revisions: | Endpoint | Use it for | | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | `GET /saved-groups-revisions` | List revisions across every saved group. Filter by `savedGroupId`, `status`, `author`, or `mine`. | | `GET /saved-groups-revisions/{savedGroupId}` | List revisions for one saved group. | | `GET /saved-groups-revisions/{savedGroupId}/latest` | The most recently updated open draft. 404 if no open draft. `mine=true` filters to your own drafts. | | `GET /saved-groups-revisions/{savedGroupId}/{version}` | Fetch a specific revision, including `baseSavedGroup`, `proposedSavedGroup`, reviews, activity log. | | `GET /saved-groups-revisions/{savedGroupId}/{version}/merge-status` | Dry-run merge against the current live state. Returns conflicts and whether they can auto-merge. | `baseSavedGroup` is the snapshot taken when the draft opened. `proposedSavedGroup` is what the live saved group would look like if the draft were merged right now — useful for previewing changes without interpreting the raw JSON Patch ops in `proposedChanges`. ### Reverting `POST /saved-groups-revisions/{savedGroupId}/{version}/revert` creates a new revision whose content matches the specified historical revision. Pass `{"strategy": "draft"}` (default) to stage the revert as a draft, or `{"strategy": "publish"}` to publish immediately. Publish obeys the same approval rules as a normal publish. ```bash curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/2/revert' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"strategy": "draft", "title": "Roll back the Q2 cohort change"}' ``` ### Discarding Open drafts you no longer want can be discarded with `POST .../discard`. Merged and already-discarded revisions are rejected. ```bash curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/discard' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"reason": "superseded by revision 5"}' ``` ## Merge Conflicts Your draft can diverge from the live saved group if someone else publishes changes while your draft is open. Publishing detects this and rejects the call with `409 Conflict`, including the conflicting fields in the response body. The recovery flow is to rebase, then re-publish: ```bash # 1. Inspect conflicts curl 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/merge-status' \ -H 'Authorization: Bearer YOUR_API_KEY' # 2. Rebase, picking a resolution strategy per conflicting field curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/rebase' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "conflictResolutions": { "values": "union", "description": "overwrite" } }' # 3. Retry publish curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/publish' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{}' ``` Per-field resolution strategies: - `overwrite` — keep the draft's value. - `discard` — keep the live value. - `union` — concatenate arrays (only valid for `values` on list saved groups). Pass `customValues` to supply your own merged array instead of the default union. `merge-status` is purely informational — it doesn't lock anything. If you need strict optimistic locking, call `merge-status`, then `publish`, and on `409` re-fetch and retry. ## Approval Flows When [Saved Group approvals are required](/features/publishing-and-approval-flows#saved-group-approval-flows), a draft must be approved before it can be published. Approval is not required when the caller has the **Bypass draft approvals** policy in every Project assigned to the Saved Group, or when the organization enables **REST API always bypasses approval requirements**. ### Requesting a Review Move a draft from `draft` to `pending-review`. Reviewers are notified per the org's approval-flow settings. ```bash curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/request-review' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{}' ``` ### Reviewing A reviewer submits a decision with `POST .../submit-review`. `decision` is one of `approve`, `request-changes`, or `comment`. With `blockSelfApproval` enabled (controlled by the org's **Require approval from a non-editor** setting), authors and contributors cannot submit `approve` on their own drafts. ```bash curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/submit-review' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"decision": "approve", "comment": "LGTM"}' ``` The revision's `reviews` array on subsequent reads shows every decision, who made it, and when. ### Publishing with Approvals Once the revision is `approved`, publish proceeds normally: ```bash curl -X POST 'https://api.growthbook.io/api/v1/saved-groups-revisions/grp_abc123/4/publish' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{}' ``` There are three ways to publish without an approved revision: 1. **Direct-write bypass** — `POST /saved-groups` and `POST /saved-groups/{id}` can send `bypassApproval: true` to update the live Saved Group without creating a revision. The caller needs **Bypass draft approvals** in every affected Project. 2. **Permission-based publish** — the revision publish endpoint automatically skips the approval check for callers with **Bypass draft approvals** in every Project assigned to the Saved Group. The `bypassApproval` field on this endpoint is accepted for backwards compatibility but does not change the result. 3. **Organization-wide REST bypass** — when **REST API always bypasses approval requirements** is enabled, REST API requests can publish without approval. ### Permissions Saved Group lifecycle permissions are independent: - **Create** creates a Saved Group. - **Edit** opens and changes drafts, requests review, rebases, and discards drafts. - **Review** approves a draft or requests changes. A plain review comment can also be added with the general **Comments** permission. - **Publish** applies a draft to the live Saved Group and unarchives it. - **Revert** restores a previously published revision. - **Archive & delete** archives a Saved Group or permanently deletes one that is already archived. - **Bypass draft approvals** skips required review and allows an out-of-date draft to be force-published. GrowthBook checks permissions again when a draft is published. If the draft changes the Saved Group's Projects, the caller must have the required permission in both the current and destination Projects. A user with only **Revert** or **Archive & delete** can still create and complete a draft that contains only that action. Adding any unrelated change requires **Edit** and **Publish** access. --- ## Stale Feature Flag Detection import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Stale Feature Flag Detection Over time, some feature flags become "stale" — they're no longer actively used or relevant, but they remain in your system. Stale flags add clutter to your dashboard and make it harder to understand which flags actually matter. GrowthBook can automatically detect these flags so you can clean them up. ## Viewing Stale Flags Stale flags are surfaced in the main feature flag table. Look for the **Stale** column, which indicates whether a flag has been marked stale. ![Stale Flag Column](/images/features/stale-ff-overview.png) You can also see the stale indicator on individual feature pages, next to the feature name. ![Stale Flag Indicator](/images/features/stale-ff-page-view.png) ## How GrowthBook Determines Stale Status Hover over the warning icon next to a stale flag to see the specific reason it was flagged. A flag is considered stale if it hasn't been updated in two weeks **and** meets at least one of these criteria: - No active environments (the flag is disabled everywhere) - One-sided rules that send 100% of traffic to a single variation ## Toggling Stale Detection You can disable stale detection for a specific flag from the feature details page or from the features table. To re-enable stale detection, navigate to the menu in the top right of the feature details page. :::tip Use [Code References](/features/code-references) alongside stale detection to see exactly where a stale flag is used in your codebase, making cleanup faster and safer. ::: --- ## Targeting Conditions import MaxWidthImage from "@site/src/components/MaxWidthImage"; Add targeting conditions to a [rule](/features/rules) to control which values a user sees. Conditions are evaluated against **attributes** you pass into the SDK — like user ID, country, plan type, or any custom property. This page covers how to set up attributes, write conditions, and use Saved Groups for reusable targeting. ## Attributes For targeting to work, you need to pass attributes into the GrowthBook SDK and define them in the GrowthBook app. Here's a quick overview of how to do both. ### Passing Attributes into the SDK Attributes are passed into your SDKs as key-value pairs. The keys are completely customizable — use whatever fits your application's data model. Here's an example from the JavaScript SDK: ```ts growthbook.setAttributes({ id: "123", email: "hello@growthbook.io", country: "US", url: window.location.href, userAgent: navigator.userAgent, admin: true, age: 50, }); ``` ### Defining Attributes in GrowthBook In addition to passing attributes into the SDK, define the same attribute keys in the GrowthBook app under **SDK Connections** → **Attributes**: ![List of targeting attributes](/images/features/edit-targeting-attributes-2.png) :::info Attribute Values Are Never Sent to GrowthBook The actual values of targeting attributes (e.g., user IDs, emails) are never sent to GrowthBook. They are only stored in memory locally within the SDK. This architecture keeps your users' PII safe and secure. ::: Each attribute has 4 parts: - **Attribute name** — How the attribute is referenced in the SDK. - **Data Type** — The type of value the attribute holds. - **Identifier** — Whether this attribute uniquely identifies a person, account, company, or device. Identifiers are used for experiment assignments. - **Projects** — Which projects the attribute is available in. If no projects are selected, the attribute is available everywhere. ### Attribute Data Types GrowthBook supports the following attribute data types: | Data Type | Description | Example | | --------------------------- | ------------------------------------------------------ | ---------------------------------- | | **Boolean** | true or false value | `true`, `false` | | **Number** | Floats or integers | `25`, `3.14`, `-8` | | **String** | Freeform text | `"alice@example.com"`, `"blue"` | | **Enum** | Short list of predefined string values | `"basic"`, `"pro"`, `"enterprise"` | | **Secure String** | String values hashed before passing to SDK for privacy | `"secure-id-123"` | | **Array of Strings** | List of strings, useful for tags or categories | `["admin", "editor"]` | | **Array of Numbers** | List of numeric values | `[1, 2, 3, 5, 8]` | | **Array of Secure Strings** | Multiple hashed string values for secure targeting | `["user123", "user456"]` | #### Constrained List Attributes Array attributes (**Array of Strings**, **Array of Numbers**, **Array of Secure Strings**) can optionally be restricted to a fixed set of allowed values. When creating or editing an array attribute under **SDK Configurations → Attributes**, fill in the **Allowed Values** field with a comma-separated list — a user can still hold multiple values at once (e.g. `["admin", "editor"]`), but each must be one of the allowed values. Once restricted, targeting conditions on that attribute use a typeahead multi-select that only accepts the allowed values (no free-form entry) and offer the `includes any of` / `includes none of` operators. Leave **Allowed Values** blank to keep the list unrestricted. #### Changing an Attribute's Data Type You can change an attribute's data type in place (including String → Enum) by editing it under **SDK Configurations → Attributes**. You do **not** need to create a new attribute or re-point features — the change is applied in place and existing targeting conditions keep evaluating, since values are stored as-is and the SDK does not enforce data types. When converting to a constrained type (Enum, or an array with Allowed Values), make sure the allowed values include every value already used in existing conditions. Conditions that reference an out-of-list value, or that use an operator no longer offered for the new type, keep running but become harder to edit. The edit modal lists the features, experiments, and condition groups that reference the attribute so you can audit them first. #### Semantic Version Targeting GrowthBook supports semantic version string comparisons, so that `1.0.10` is correctly treated as greater than `1.0.9`. To use this, create or edit a String attribute under **SDK Configurations** → **Attributes** and select **Version string** in the format dropdown. ![Version string attribute](/images/features/targeting-semantic-versions-2.png) After saving, targeting operators (e.g., `is greater than`) automatically use a version-safe comparison function. :::warning SDK Compatibility Semantic version targeting is only supported in some SDKs. Check the release notes for your specific SDK to confirm compatibility. ::: #### Date Targeting GrowthBook supports a date format for string attributes that makes it easier to target by date. To use this, create or edit a string attribute under **SDK Configurations → Attributes** and select **Date string** in the format dropdown. ![Date string attribute](/images/features/targeting-dates-2.png) After saving, certain targeting operators (e.g., `is after or on` or `is equal to`) display a date picker input. Dates entered with the date picker are saved as ISO-formatted date strings (e.g., `2024-07-23T20:06`). #### Country Code Targeting Use [2-character ISO country codes](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) to simplify targeting by country. To set up country code targeting: - Create an attribute and set the **Data Type** to `String`. - Change the **String Format** to `ISO Country Code (2 letter)`. - Save the attribute. When setting targeting rules, you'll get a dropdown to select a country directly. ### Defining Conditions GrowthBook provides a visual UI for defining targeting conditions using your attributes. ![Simple targeting conditions](/images/features/targeting-simple-2.png) By default, string attributes are _case sensitive_. For example, if you pass `us` into the SDK, it will not match `US`. To make string attributes case insensitive, enable the **Case insensitive** toggle: ![Case insensitive toggle](/images/features/feature-case-insensitive-toggle.png) ### Advanced Mode For more advanced targeting, enter conditions as JSON by clicking **Advanced Mode**. The JSON structure uses a MongoDB-inspired query syntax. Multiple conditions are always joined with `AND` (except when explicitly using `$or`/`$nor`). Below are all supported operators with examples. #### Simple Equality Key/value pairs for exact matches: ```json { "attribute1": "value1", "attribute2": 123, "attribute3": false } ``` #### Comparison Operators Basic comparison operators for string/number attributes: - `$eq` (equals) - `$ne` (not equals) - `$lt` (less than) - `$lte` (less than or equal to) - `$gt` (greater than) - `$gte` (greater than or equal to) - `$regex` (regular expression match, string attributes only) - `$in` (in array) - `$nin` (not in array) ```json { "foo": { "$gt": 10, "$lte": 99 }, "bar": { "$in": ["a","b","c"] }, "baz": { "$regex": "^test-([0-9]+)$" } } ``` #### Semantic Version Operators Comparison operators for semantic version strings: - `$veq` (equals) - `$vne` (not equals) - `$vlt` (less than) - `$vlte` (less than or equal to) - `$vgt` (greater than) - `$vgte` (greater than or equal to) ```json { "appVersion": { "$vgt": "1.5.6", "$vlte": "5.4.0" } } ``` #### Array Operators Operators for array attributes: - `$elemMatch` (at least one element must match the specified condition) - `$all` (all of the specified values must exist in the array) - `$size` (array length must match the specified condition) ```json { "emails": { "$elemMatch": { "$regex": "@gmail.com$" } }, "hobbies": { "$all": ["hiking","tennis","chess"] }, "tags": { "$size": { "$gt": 5 } } } ``` #### Miscellaneous Operators - `$exists` (tests if the attribute value is null or not) - `$type` (tests if the attribute's type matches the type specified) - `$not` (inverts a nested condition) ```json { "alternateEmail": { "$exists": true }, "foo": { "$type": "string" }, "name": { "$not": { "$regex": "^J" } } } ``` #### Logical Operators Logical operators with arbitrary nesting levels: - `$or` - `$nor` - `$and` - `$not` ```json { "$or": [ { "$not": { "foo": "abc" } }, { "$and": [ {"bar": true}, {"baz": 123} ] } ] } ``` :::info MongoDB-style Query Syntax GrowthBook uses MongoDB query syntax because it is easy to read, write, and well documented. Conditions are never executed against a database — the SDKs include a lightweight interpreter for this syntax that runs entirely locally. ::: ## Saved Groups Saved Groups let you target the same group of users across multiple features and experiments. Define a group once and reuse it everywhere — for example, beta testers or high-value customers. ![Saved groups overview](/images/features/saved-groups-overview.png) ### Condition Groups Define targeting rules based on user attributes. For example, target users who are located in the US and on a mobile device. ![Saved groups condition, showing how to add a US user on a mobile device](/images/features/saved-groups-condition.png) ### ID Lists Manually define targeted users via text input or by uploading a CSV. For example, create a beta testers group by uploading a CSV of user IDs. ![Saved Group UI, focusing on adding an ID list](/images/features/saved-groups-id-list.png) ID Lists are limited to 1 MB. Using multiple large ID Lists may impact performance as it increases the SDK payload. Keep ID Lists as concise as possible. ID Lists accept the following attribute types: `string`, `secureString`, or `number`. For targeting other types of attributes, use [Condition Groups](#condition-groups). :::tip Optimized ID lists Enterprise users can use optimized ID lists to reduce payload size. Enable this feature by editing your SDK connection and toggling on **Pass ID Lists by reference**. If the toggle isn't visible, your SDK is not compatible with this feature. ::: For more advanced targeting based on the state of other feature flags, see [Prerequisite Features](/features/prerequisites). --- :::note Legacy behavior for empty ID lists Empty lists would previously be ignored in the SDK payloads, causing targeting conditions referencing those lists to always evaluate to true. ID Lists created after the behavior was changed properly use the empty list, so rules checking whether a value is in the list always evaluate to false. Lists created before the change preserve the old behavior so as not to break existing features. To make an ID list with the legacy behavior evaluate to false instead of true, add a single placeholder value such as "" or "\_" so that the list isn't empty. ::: --- ## GrowthBook and Contentful GrowthBook and Contentful work together to enable content teams to run A/B tests without requiring developer intervention for each test. Once configured, content editors can create and manage experiments directly within the Contentful interface, eliminating the need for code changes. With these experiments, teams can leverage GrowthBook's advanced experimentation analysis to understand outcomes and make informed product decisions. ## Requirements Before you begin, ensure you have the following: - [GrowthBook account](https://www.growthbook.io/ "https://www.growthbook.io") (free tier available) - [Contentful account](https://www.contentful.com/ "https://www.contentful.com") (free tier available) ## Key concepts ### Experiment content type Installing GrowthBook's Contentful plugin adds a new **GrowthBook Experiment** content type to your Contentful space. This content type lets you create GrowthBook experiments right from the Contentful interface. ### Variations field The **GrowthBook Experiment** includes a **Variations** field that acts as a container for the content you want to test. For example, to test which featured product increases sales the most, you'd add these products to the **Variations** container. ![Contentful entry with GrowthBook experiment variations](/images/guides/contentful-experiment.png) ### Experiment Once configured, you're ready to start your experiment. This syncs the data created in Contentful with GrowthBook, so you can analyze the experiment results and see which version of your test performs best. The diagram below illustrates how the above components work together. ![Contentful and GrowthBook flow diagram](/images/guides/contentful-growthbook-diagram.png) Continue reading to learn how to set up GrowthBook and Contentful to run A/B tests on your site. ## Set up GrowthBook ### 1. Create an SDK Connection An SDK connection is a link between your GrowthBook account, your frontend codebase, and the Contentful plugin. It allows GrowthBook to serve the correct content variations to your users. 1. Log in to GrowthBook 2. Navigate to **SDK Configuration** → **SDK Connections** in the sidebar 3. Click **Add SDK Connection** 4. Select **JavaScript** or **React** as your language 5. Save your **API Host** URL for later use ![GrowthBook SDK connection](/images/guides/contentful-sdk-connection-data.png) ### 2. Configure data source A data source is the connection between GrowthBook and your analytics platform. This connection allows GrowthBook to analyze your experiment data and provide insights. It's also informs Contentful which data source to use when creating experiments. 1. Go to **Metrics and Data** → **Data Sources** in GrowthBook 2. Click **Add Data Source** 3. Choose your preferred analytics platform 4. Follow the connection guide 5. Save your data source ID (found in the URL of your data source page) ![GrowthBook data source ID location](/images/guides/contentful-data-source-id.png) ### 3. Generate API key To connect Contentful to GrowthBook, you need an API key with admin permissions. This key allows Contentful to communicate with GrowthBook and sync your experiments. 1. Navigate to **Settings** → **API Keys** 2. Click **Create New Key** 3. Select `admin` role 4. Save the generated key securely ![GrowthBook API test](/images/guides/contentful-api-keys.png) ## Install and configure the GrowthBook app :::tip Find the [official GrowthBook app](https://www.contentful.com/marketplace/growthbook-experiment/) in the Contentful marketplace. ::: 1. Go to **Apps** → **Marketplace** 2. Search for GrowthBook 3. Click **Install** 4. Configure the plugin: - Enter your GrowthBook API host URL from above - Enter your data source ID from step 2 - Enter your Admin API key from step 3 - Click **Save** ## Run your first experiment With everything configured, you're ready to run your first experiment. ### 1. Add or update your content model Create a new content model or update an existing one to include the **GrowthBook Experiment** content type. This content type will allow you to create and manage experiments directly within Contentful. ### 2. Add the **GrowthBook Experiment** to a page Open the content entry where you want to add the experiment. In the relevant reference field, click **Add content** and choose **GrowthBook Experiment**. ### 3. Add **Variations** Name your experiment. Add your variations by linking existing Contentful entries or creating new ones. ### 4. Launch experiment Click **Create New Experiment** and then **Start Experiment**. This will sync your variations and experiment data to GrowthBook and start your A/B test. ### 5. Publish changes Publish changes to the **Experiment** content as well as to the page that contains the experiment. Your changes will now be live, serving different content to users! ## Frontend integration To accommodate the **GrowthBook Experiment** type, changes need to be made to your front-end codebase. At a high level, a front-end integration requires the following parts: ### Connect the GrowthBook SDK [Adding the GrowthBook SDK to your codebase](/lib/) enables the content variations returned by Contentful to be properly rendered: `User 123` sees `Product A`, while `User XYZ` sees `Product B`. ### Content Model Structure The **GrowthBook Experiment** content model is fetched via the following GraphQL code, which contains the `featureFlagId` and the `variationsCollection`. The response contains data about your experiment content, as you set it up in Contentful. To get specific Variation content, make an additional call to the Contentful API using its `id` and `_typename`, or, as in the example below, add the fields for each Content Type you want to be able to experiment on. ```ts // Example GraphQL fields for your content type export const GROWTHBOOK_EXPERIMENT_GRAPHQL_FIELDS = ` sys { id } featureFlagId variationsCollection { items { sys { id } __typename ... on yourContentType { ${YOUR_CONTENT_TYPE_GRAPHQL_FIELDS} } } } `; ``` ### Variation Selection Logic The `getVariation` function uses your GrowthBook data to determine the variation's state. Calling `gb.getFeatureValue` with the `featureFlagId` returns the index of the variation to show in the `variationsCollection`. ```ts // Example getVariation function in TypeScript export function getVariation( gb: GrowthBook, growthbookExperiment: GrowthbookExperimentInterface ) { const featureFlagId = growthbookExperiment.featureFlagId; const variationsCollection = growthbookExperiment.variationsCollection; const index = gb.getFeatureValue(featureFlagId ?? "", 0); if (index > variationsCollection.items.length + 1) { return variationsCollection.items[0]; } return variationsCollection.items[index]; } ``` ### Tracking callback An essential part of configuring the GrowthBook SDK to run experiments is setting up the tracking callback. It's what sends the experiment exposure data (that `User 123` saw `Product A`) to your data warehouse, which, in turn, enables GrowthBook to analyze your experiment and determine which variation performed best. Your particular implementation will depend on your codebase and will require a developer's help. The good news is that this change only needs to be made once to enable streamlined A/B testing directly from Contentful. We also have a [full example project](https://github.com/growthbook/examples) that shows how to integrate Contentful and GrowthBook with Next.js. ## FAQs and troubleshooting ### Common questions **Q: How does the GrowthBook app affect my Contentful environment?** A: The app creates a new content type called GrowthBook Experiment that enables A/B testing functionality. **Q: Where can I activate my experiments?** A: Experiments are managed through both Contentful (content variations) and GrowthBook (experiment settings and analysis). **Q: What happens after an experiment is complete?** A: Replace the GrowthBook Experiment content with the winning variation. ### Troubleshooting **Variations not showing up?** - Ensure all variation content is published - Verify the experiment is properly configured in Contentful and GrowthBook - Check that the frontend integration is correctly implemented **Configuration not saving?** - Make sure to click the "Save" button after making changes - Verify you have the necessary permissions in both platforms --- ## GrowthBook and BigQuery # GrowthBook and Create React App ## 1. Set up Create React App Create React App is simple to get a new project started. This guide does the standard process with one extra command to install the GrowthBook SDK and the nanoid library: ```bash npx create-react-app my-app cd my-app npm install --save @growthbook/growthbook-react nanoid npm start ``` Then open `http://localhost:3000/` and make sure the app is working ## 2. Set up GrowthBook In this tutorial assume you are using the cloud-hosted version of GrowthBook, which is free for small teams, but you can also use the open source version and host it yourself if you prefer. :::note GrowthBook uses the concept of Feature Flagging to run A/B tests. Basically, you wrap the code you want to test in a conditional check `if (feature.on) ...` and then you run an A/B test within GrowthBook to turn the feature on for 50% of users and off for the other 50% (or whatever percentage you like). ::: To start, go to https://app.growthbook.io and register a new account. Then there are a couple steps required before you can run an experiment. ## 3. Install and configure the SDK Next, click on Step 1: Install our SDK and you should see API keys for dev/production as well as sample code. Since you already ran the npm i command at the start, you can skip that part. I'll walk through the different parts below: First, in `src/index.js`, import the GrowthBook SDK and nanoid library: ```ts // ... after existing imports import { GrowthBook, GrowthBookProvider } from "@growthbook/growthbook-react"; import { nanoid } from "nanoid"; ``` Then you will need to generate an anonymous visitor id, which is used to assign an A/B test variation to a user. In this example we'll persist this id in localStorage so if the user refreshes our app they will get assigned the same variation as before. ```ts let visitor_id = localStorage.getItem("visitor_id"); if (!visitor_id) { visitor_id = nanoid(); localStorage.setItem("visitor_id", visitor_id); } ``` Then, you create a GrowthBook instance with our visitor id and a tracking callback when a user is put into an experiment. ```ts const growthbook = new GrowthBook({ attributes: { id: visitor_id, }, trackingCallback: (experiment, result) => { console.log({ experimentId: experiment.key, variationId: result.variationId, }); }, }); ``` After that, you can fetch the list of features from the GrowthBook API and pass them into the SDK: ```ts const FEATURES_ENDPOINT = "https://cdn.growthbook.io/api/features/..."; fetch(FEATURES_ENDPOINT) .then((res) => res.json()) .then((json) => { growthbook.setFeatures(json.features); }); ``` Make sure to swap out the `FEATURES_ENDPOINT` constant above with your own dev API key you see in the GrowthBook application. Lastly, you'll need to wrap the app in a GrowthBookProvider component which will let us run A/B tests from anywhere in the app. ```tsx ReactDOM.render( , document.getElementById('root') ); ``` ## 4. Create and use a feature Now that the SDK is installed and fully integrated in our application, you can finally create the `show-logo` feature. Back in GrowthBook, Click on Step 2 of the quick start instruction or click on add new feature. Fill in the following info: - Feature key: `show-logo` - Dev: toggle on - Prod: toggle off - Value Type: boolean (on/off) - Behavior: A/B Experiment - Tracking Key: `show-logo` - Sample Users based on attribute: `id` - Variations and Weights: leave default (OFF/ON, 50/50 split) - Fallback Value: `OFF` There's a lot of fields there, but hopefully it's pretty straight forward what's happening. We setup a new boolean feature called show-logo, that's only enabled in dev and running an A/B test where 50% get ON and 50% get OFF Now you can switch back to our React app and reference this feature in our code. In src/App.js, we currently have the following code: ```html ``` Let's add an import at the top of the file: ```js import { IfFeatureEnabled } from "@growthbook/growthbook-react"; ``` And wrap the img element in an IfFeatureEnabled component: ```tsx ``` Now, if you refresh your app, the A/B test should be running! If you're part of the lucky 50% that are in the B variation (no logo), it should be pretty obvious. If you happen to be in the A variations, you can verify you're in the test by looking in DevTools for our trackingCallback console.log. You can test out different variations by deleting the visitor_id from localStorage and refreshing your app. Repeat a few times and you should see each version of the page about half of the time. If you want an easier and faster way to QA the variations, you can download the GrowthBook DevTools Browser Extension. ## 5. Analyze Results Before you can analyze the results, you will need to connect GrowthBook to the event tracking and a data source. In the trackingCallback in `src/index.js`, instead of doing a `console.log`, you could use [Mixpanel](/guide/mixpanel), Rudderstack, Jitsu, Segment or another event tracking system. Then, throughout your app, you can similarly track events when users do something you care about, like sign up, convert, or buy something. Once you do that, GrowthBook can connect to your event tracking system, query the raw data, run it through a stats engine, and show you the results. Follow the directions for the data source you're using. ## Next Steps There's so much more you can do with GrowthBook beyond a simple on/off A/B test... - Add complex targeting and rollout rules for your features - Read the full [React SDK Docs](/lib/react) for more details and ways to use feature flags - Install the [Chrome](https://chrome.google.com/webstore/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/) DevTools Browser Extension to test different variations and scenarios - Read about the [powerful statistics engine](/statistics/overview) that is used to analyze experiment results. --- ## Feature Flags and A/B Tests with Deno, Hono, and GrowthBook # Feature Flags and A/B Tests with Deno, Hono, and GrowthBook :::note The only requirement for this tutorial is to [install Deno](https://docs.deno.com/runtime/getting_started/installation/). The installation process is straightforward, and you'll be ready to follow along easily. ::: ## Introduction In this guide, we’ll use **Deno**, **Hono**, and **GrowthBook** to build a landing page for a fictional FinTech app called **Bayes Bank**. With this setup, we can use feature flags and A/B testing on a server-rendered landing page to test and deliver different experiences to users. All tutorial code is available in our [Examples repo](https://github.com/growthbook/examples/tree/main/deno-hono). ![Bayes Bank Homepage, example site for the Deno, Hono, GrowthBook demo](/images/guides/deno-homepage-demo.png) ### What are Deno, Hono, and GrowthBook? - [**Deno**](https://deno.com/) is a JavaScript and TypeScript runtime developed by Ryan Dahl (creator of Node.js) that simplifies the development process and resolves many limitations from Node. - [**Hono**](https://hono.dev/) is a fast, lightweight web framework that works like an updated, modernized Express, optimized for Deno and other runtimes. - [**GrowthBook**](https://growthbook.io/) (that’s us!) is an open-source platform for feature flags and A/B testing. We offer a user-friendly interface for creating and managing flags, running tests, and analyzing results. In this tutorial, learn how to use these technologies to build a feature-flagged landing page. Here’s a diagram that shows how these technologies fit together: ![Diagram showing Deno, Hono, and GrowthBook](/images/guides/deno-diagram.png) ## Project setup ### Initialize Hono Run the following command to create a new Hono project, selecting **Deno** as your runtime. (Optionally, substitute `my-app` for your project’s name.) ```bash deno run -A npm:create-hono@latest my-app ``` This command creates a new directory with the following files: - **`README.md`**: Contains basic instructions for starting the server. - **`deno.json`**: Defines project dependencies and configuration. - **`main.ts`**: The core file where your application logic will reside. Let’s take advantage of the fact that [Hono can support JSX](https://hono.dev/docs/guides/jsx) out of the box and rename `main.ts` to `main.tsx`. ### Install dependencies Next, install **GrowthBook** and **Tailwind CSS**: ```bash deno add jsr:@growthbook/growthbook npm:tailwindcss ``` The `deno add` command registers dependencies via your import map in `deno.json`. The prefix before package names indicates which registry the package comes from. :::tip GrowthBook on the JSR The GrowthBook JS SDK is now available on the JSR. Learn more in our [announcement post](https://blog.growthbook.io/announcing-growthbook-on-jsr/). ::: ## Configure Tailwind Generate a Tailwind config: ```bash deno run -A npm:tailwindcss init -p ``` This creates `postcss.config.js` and `tailwind.config.js`. Update the Tailwind config to target the right files: ```js /** @type {import('tailwindcss').Config} */ module.exports = { content: ["./**/*.{js,ts,jsx,tsx}"], theme: { extend: {}, }, plugins: [], }; ``` Create a `src/css` folder and add a new file called `input.css`. Add Tailwind directives to generate base styles: ```css @tailwind base; @tailwind components; @tailwind utilities; ``` ### Configure development scripts Add the following tasks to `deno.json` to streamline development: ```json { "imports": { "@growthbook/growthbook": "jsr:@growthbook/growthbook@^1.2.2", "hono": "jsr:@hono/hono@^4.6.8", "tailwindcss": "npm:tailwindcss@^3.4.14" }, // highlight-start "tasks": { "start": "deno run --allow-net main.tsx", "watch:jsx": "deno run -A --watch main.tsx", "watch:css": "deno run -A npm:tailwindcss -i src/css/input.css -o static/style.css --watch", "dev": "deno task watch:css & deno task watch:jsx" }, // highlight-end "compilerOptions": { "jsx": "precompile", "jsxImportSource": "hono/jsx" } } ``` Running `deno task dev` will watch for file changes, automatically restart the server, and recompile CSS. ## Building the app layout To build the landing page, create a folder in `src` called `components`. Add the following components for the page layout: `Header.tsx`, `Hero.tsx`, `Copy.tsx`, and `Footer.tsx`. All of these components are available in the [demo repo](https://github.com/growthbook/examples/tree/main/deno-hono/src/components). Here’s the `Footer` component as an example: ```tsx title="/src/components/Footer.tsx" import { Props } from "../../main.tsx"; export const Footer = ({ companyName }: Omit) => { return ( © 2024 {companyName}. All rights reserved. Terms of Service Privacy ); }; ``` And here’s the rendered component: ![Rendered footer component](/images/guides/deno-footer-component.png) ### Main server code The code generated when initializing Hono instantiates a new app, defines a route handler for the homepage, and starts the server. ```tsx title="main.tsx" import { Hono } from 'hono'; const app = new Hono(); app.get('/', (c) => c.text('Hello Hono!')); Deno.serve(app.fetch); ``` Now, import the `Navbar` and `Footer` components and build a layout component for reuse: ```tsx title="main.tsx" import type { PropsWithChildren } from "hono/jsx"; import { Navbar } from "./src/components/Navbar.tsx"; import { Footer } from "./src/components/Footer.tsx"; export function Layout({ companyName, children }: PropsWithChildren) { return ( Deno + Hono + GrowthBook {children} ); } ``` Add `Hero.tsx` and `Copy.tsx` components to define the page content, then create the `Page` component to bring it all together. ```tsx title="main.tsx" const Page = ({ companyName }) => ( ); ``` Finally, modify the route to serve this page: ```tsx title="main.tsx" app.get("/", (c) => c.html()); ``` Find the final version of the `main.tsx` file in the [demo repo](https://github.com/growthbook/examples/blob/main/deno-hono/main.tsx). ## Integrating GrowthBook Create a file called `growthbookMiddleware.ts` to use feature flags and A/B tests in the app: ```ts title="growthbookMiddleware.ts" import { GrowthBook } from "@growthbook/growthbook"; import { getCookie, setCookie } from "hono/cookie"; import { createMiddleware } from "hono/factory"; export const growthbookMiddleware = createMiddleware(async (c, next) => { // Instantiate GrowthBook with your connection details const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", // Update with your API host clientKey: "YOUR-SDK-KEY", trackingCallback: (experiment, result) => { console.log("Experiment Viewed", { experiment, result }); // Update function to send experiment exposure to your analytics platform }, }); // Get a preexisting cookie (ensures users are kept in the same variant) let uuid = getCookie(c, "gb_uuid"); // Create the UUID/cookie if it doesn't exist if (!uuid) { uuid = crypto.randomUUID(); setCookie(c, "gb_uuid", uuid); } // Pass GrowthBook the ID and other attributes as needed gb.setAttributes({ id: uuid, // ... }); // Initialize GrowthBook await gb.init({ timeout: 1000 }); // Add the instance to the context c.set("gb", gb); // Continue the req/response await next(); // Clean up once the request closes gb.destroy(); }); ``` Add this middleware to your route in `main.tsx`: ```tsx title="main.tsx" import { growthbookMiddleware } from "./growthbookMiddleware.ts"; app.get("/", growthbookMiddleware, (c) => { const gbInstance = c.var.gb; return c.html(); }); ``` ## Using feature flags in the app Pass `gb` to components to conditionally render elements based on feature flags. For Bayes Bank, we want to test different headlines on the landing page to see if they boost conversions. Update the text in `Hero.tsx` to be handled by a feature flag: ```tsx title="/src/components/Hero.tsx" export function Hero({ gb }: Props) { const headline = gb.getFeatureValue("headline", "Revolutionize Your Finances with AI"); return {headline}; } ``` GrowthBook’s `getFeatureValue` method takes the flag’s ID (`headline`) and a fallback value, in case the app can’t reach GrowthBook’s API. Now, different headlines will appear based on how the feature flag is configured in GrowthBook. ### Configuring GrowthBook In GrowthBook, create a feature flag called `headline` and set its type to “String” with a default value of “Revolutionize Your Finances with AI.” ![GrowthBook feature flag configuration](/images/guides/deno-growthbook-feature-flag.png) Next, click **Add Rule** → **Experiment** to set up an A/B test. ![GrowthBook A/B test configuration](/images/guides/deno-growthbook-abtest.png) Set variation `0` to the default value, “Revolutionize Your Finances with AI,” and variation `1` to "Save Money and Grow Your Wealth with AI." Click **Save** to start the experiment 🧪 Below, check out the control and variation, all rendered server side! ![Rendered A/B test control](/images/guides/deno-control.png) ![Rendered A/B test variation](/images/guides/deno-variation.png) ## Summary In this tutorial, we explored how to build a dynamic landing page for **Bayes Bank** using **Deno**, **Hono**, and **GrowthBook**. By leveraging these modern technologies, we created a server-rendered application that incorporates feature flags and A/B testing, enabling us to test different user experiences seamlessly. With the fundamentals covered, you now have the tools to expand on this project, experiment with additional features, and fine-tune the user experience for Bayes Bank. ## Resources - [Tutorial codebase](https://github.com/growthbook/examples/tree/main/deno-hono) --- ## Express.js and GrowthBook import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Feature Flags with Express.js and GrowthBook This guide walks through how to add GrowthBook feature flags to an [Express.js](https://expressjs.com/) application. It assumes you are starting from scratch, so if you already have an Express.js application, you can skip to step 2. ## 1. Create an Express.js app First, install Express.js ```bash npm2yarn npm install express ``` Then, create an `index.js` file with a simple hello world route ```js const express = require('express') const app = express() app.get('/', (req, res) => { res.send('Hello World!') }) // Listen on port 3000 const port = 3000 app.listen(port, () => { console.log(`Example app listening on port ${port}`) }) ``` Finally, run the app with: ```bash node index.js ``` Visit `http://localhost:3000` and you should see a "Hello World!" response! ## 2. Create a GrowthBook Account GrowthBook can be [self-hosted](/self-host), but for simplicity, we will use GrowthBook Cloud in this guide, which is free for simple projects like this. Go to [https://app.growthbook.io](https://app.growthbook.io) and create a new account if you don't have one yet. Once you are logged in, create an **SDK Connection** and select Node.js as the language. This will generate a unique **Client Key** for you. Keep note of this key as we will use it in the next step. ## 3. Integrate the GrowthBook JavaScript SDK into the Express.js app Create the file `.env` if it doesn't exist yet and add the generated key there: ```bash GROWTHBOOK_API_HOST=https://cdn.growthbook.io GROWTHBOOK_CLIENT_KEY= ``` Now let's install the GrowthBook JavaScript SDK ```bash npm2yarn npm install @growthbook/growthbook ``` Lets now modify the top of `index.js` file and create a **GrowthBookClient** instance: ```js const express = require('express') const app = express() const { GrowthBookClient } = require("@growthbook/growthbook") // GrowthBookClient instance const client = new GrowthBookClient({ apiHost: process.env.GROWTHBOOK_API_HOST, clientKey: process.env.GROWTHBOOK_CLIENT_KEY }); // Initialize it client.init().then((status) => { console.log("GrowthBook initialized", status); }); ``` Lets also add a middleware to `index.js` that creates a user-scoped instance for every request. Make sure to place this above the route handlers: ```js app.use((req, res, next) => { // Attributes about the current user/request const userContext = { attributes: { // In a real app this would come from a cookie or session // We would also add more attributes like country, etc id: "123", } } // Make this available to all subsequent route handlers req.growthbook = client.createScopedInstance(userContext); next(); }); ``` Lets re-start the Node process and make sure you see the "GrowthBook initialized" message in the console and the status is successful. Since we are using a `.env` file, we need to modify the command to tell Node.js to load it. ```bash node --env-file=.env index.js ``` ## 4. Create a Feature in GrowthBook Back in the GrowthBook application, we can create a new feature. For this tutorial, we'll make a simple on/off feature flag that determines whether or not we show the message in Spanish. ![GrowthBook Create Feature](/images/guides/express-js-1-create-feature.png) The key we chose (spanish-greeting) is what we will reference when using the GrowthBook SDK. We can now edit the route in `index.js` to use this flag: ```js app.get('/', (req, res) => { let message = "Hello World!"; if (req.growthbook.isOn("spanish-greeting")) { message = "Hola Mundo!"; } res.send(message); }) ``` Now, if you restart the Node process and refresh the page in the browser, you will still see the original "Hello World!" because when we created the feature, we set it to be Off by default. ## 5. Target Specific Users Now we can add rules to the feature to turn it on for specific users. In the `userContext` we added, we hard-coded an id of "123". We can now use this id to create a rule in GrowthBook to turn on the feature for this user. On the feature page, click the "Add Rule" button and select "Force Value" as the rule type. Then add targeting by attribute and save the rule. It should look something like this: ![GrowthBook Targeting Rule](/images/guides/express-js-2-targeting-rule.png) Rules start out in a draft state in GrowthBook. You need to publish the draft to make it live. After publishing, restart the Node process once more and refresh the page. You should now see the message in Spanish. ¡Qué bueno! Try changing the id in `userContext` to something else like `456`, restart Node, and the response will now switch back to English. ## Conclusion and Next Steps In this tutorial, you learned how to use a simple feature flag in an Express app and target individual users. But this barely scratches the surface of what you can do with GrowthBook. Here are a few next steps you can take: - Use more [advanced targeting](/features/targeting) - Run [A/B tests](/features/feature-flag-experiments) View the [full Node.js docs](/lib/node) for more information on all of the options available in the GrowthBook SDK, including streaming updates, persistent caching, and more. --- ## How to A/B Test and Feature Flag with Google Tag Manager (GTM) and GrowthBook # Using GrowthBook with Google Tag Manager (GTM) ## Overview This guide shows you how to integrate GrowthBook with Google Tag Manager (GTM) to run A/B tests and manage feature flags without having to directly modify your app's code. This setup is ideal for: - Marketing teams managing website experiments - CRO agencies handling multiple client websites - Teams without direct access to website code ### Prerequisites Before starting, ensure you have: - Access to your [Google Tag Manager account](https://tagmanager.google.com/) - A [GrowthBook account](https://app.growthbook.io/) - Basic familiarity with GTM's interface ## 1. Set Up GrowthBook SDK Connection - In GrowthBook, navigate to **SDK Configuration** → **SDK Connections** - Click **Add SDK Connection** - Add a **Name**, choose **HTML Script Tag**, and click **Save** - Copy the `data-client-key` value from the script tag. You'll need this for the next step ![GrowthBook SDK Connection](/images/guides/gtm-sdk.webp) ## 2. Create GTM Tag for GrowthBook - Open your GTM workspace - Click **Tags** in the left sidebar - Click **New** to create a tag - Click **Tag Configuration** and use **Custom HTML** as the tag type - Paste the following script tag, replacing `YOUR_CLIENT_KEY_HERE` with the value from the previous step: ```html (function(s) { s=document.createElement('script'); s.async=true; s.dataset.clientKey="YOUR_CLIENT_KEY_HERE"; s.src="https://cdn.jsdelivr.net/npm/@growthbook/growthbook/dist/bundles/auto.min.js"; document.head.appendChild(s); })(); ``` - Click **Triggering** and choose the pages where you want to run this tag. Use **All Pages** to load it everywhere - Click **Save** and call your tag "GrowthBook SDK" or similar ![GTM Tag for GrowthBook](/images/guides/gtm-growthbook-sdk-custom-tag.webp) Be sure to click **Submit** and **Publish** to make your changes live. GrowthBook is now connected to your website via GTM 🤝 ## 3. Use Feature Flags and A/B Experiments It's possible to use GrowthBook in your GTM setup. This allows you to control feature flags and experiments without needing to modify your website code. :::tip If you're planning to use the [Visual Editor](/app/visual), [URL redirects](/app/url-redirects), or to [run experiments in code](/features/feature-flag-experiments), skip ahead to the next section ⏭️ ::: Create a new boolean feature flag with an A/B test. For this tutorial, we'll use a feature flag called `show-discount` to run an A/B test to show and hide an item's discount. New to feature flags in GrowthBook? Learn more about [creating them and A/B tests](/features/basics). Next, we'll use GTM to apply the feature flag to your website: - Create another Custom HTML tag in your workspace - Set the firing trigger to the pages you want to run this on - Use the following code as a starting point: ```html // Wait for the GrowthBook SDK to load before running window.growthbook_queue = window.growthbook_queue || []; window.growthbook_queue.push(function(gb) { // Function that uses feature flags to make changes to the page function applyFeatureFlags() { if(gb.isOn("show-discount")) { var priceEl = document.getElementById("price"); if (!priceEl) return; priceEl.classList.add("show-discount"); } } // Call your function initially plus whenever new data is received applyFeatureFlags(); document.addEventListener("growthbookdata", applyFeatureFlags) }); ``` In this example, we're using the `show-discount` feature flag to run an A/B experiment. When the experiment evaluates to `true`, we add a CSS class to the price element on the page. As seen in the image below, the control element shows the price, while the variation shows the original price with a strikethough and the discounted price in red. We can run this entire experiment straight from GTM, without needing to touch the website code! ![A control element with the image and name of a couch, showing just a price, and a variation element with the same image and name, showing the original price with a strikethrough and the discounted price in red](/images/guides/gtm-ab-test.webp) ## 4. Tracking via DataLayer and GTM To track experiment views, we use the `dataLayer` to capture experiment data. Whenever an experiment is viewed, we send an event with data like this: ```json { "event": "experiment_viewed", "experiment_id": "...", "variation_id": "..." } ``` In this tutorial, we'll send this data to Google Analytics 4 (GA4), which allows you to analyze experiments and their impact on your business metrics in GrowthBook. (It's also possible to forward this event to your analytics tool of choice.) :::tip You'll need to connect GrowthBook to BigQuery to analyze your experiment data. See our guide on [configuring GrowthBook with BigQuery](/guide/bigquery) for more information. ::: ### Create DataLayer Variables To capture experiment data from the `dataLayer`, create 2 new variables in GTM: - Go to **Variables** in the left sidebar - Click **New** under **User-Defined Variables** - Create 2 variables: - **Experiment ID** - Click **Variable Configuration** - Select **Data Layer Variable** - Enter `experiment_id` as the **Data Layer Variable Name** - Name the variable something descriptive like "Experiment ID" - Click **Save** - **Variation ID** - Repeat the these steps but use `variation_id` as the **Data Layer Variable Name** and call it something like "Variation ID" ![DataLayer Variables](/images/guides/gtm-data-layer-variables.webp) ### Create Custom Event Trigger This custom trigger tells GTM when to fire the GA4 tag based on the `experiment_viewed` event. - Go to **Triggers** in the left sidebar - Click **New** and **Trigger Configuration** - Choose **Custom Event** as the trigger type - Name the trigger "Experiment Viewed Event" or similar - Enter `experiment_viewed` as the **Event Name** - Set the trigger to fire on **All Custom Events** - Click **Save** ![Custom Event Trigger](/images/guides/gtm-custom-trigger.webp) ### Create GA4 Configuration Tag Let's connect GTM to GA4. (Skip this step if you've already configured GA4 in your GTM container.) - Create a new tag and select **Google Analytics** as the type - Choose **Google Tag** - Enter your [GA4 Measurement ID](https://support.google.com/analytics/answer/12270356?hl=en) - Add **Initialization - All Pages** for the trigger - Name the tag something like "GA4 Configuration" and save it ![GTM GA4 Configuration Tag](/images/guides/gtm-ga4-config.webp) ### Create GA4 Event Tag Last step! Create a tag that sends the `experiment_viewed` event data to GA4. - Create a new tag in GTM - Select **Google Analytics: GA4 Event** - Configuration: - Measurement ID: Use your GA4 ID (or reference the GA4 Configuration tag) - Event Name: `experiment_viewed` - Event Parameters: - Event Parameter: `experiment_id` - Value: `{{Experiment ID}}` - Event Parameter: `variation_id` - Value: `{{Variation ID}}` - Triggering: Select the **Experiment Viewed Event** trigger - Name the tag (e.g., "GA4 - Experiment Viewed Event") - Save the tag ![GTM GA4 Event Tag](/images/guides/gtm-ga4-event.webp) ### Testing Your Setup 1. In GTM: - Click **Preview** in the top right - Enter your website URL - Click **Start** 2. In Preview Mode: - Look for the GrowthBook SDK tag firing - Verify the `experiment_viewed` event appears - Check that the event parameters contain the correct experiment and variation IDs ![GTM Preview Mode](/images/guides/gtm-preview.webp) 3. In GA4 (may take 24-48 hours to appear): - Go to Real-Time reports - Look for the `experiment_viewed` event - Verify the parameters are being received correctly Remember to save and publish your changes in GTM once you're ready to go live. You can now track experiment views in GA4 using GTM. ## Conclusion You've successfully integrated GrowthBook with Google Tag Manager to run A/B tests and manage feature flags on your website. This setup allows you to control experiments without needing to modify your website code directly ✊ ### Next Steps - Check out our guide for connecting [GrowthBook to GA4 and BigQuery](/guide/GA4-google-analytics) - For other analytics tools, see our guide on [connecting to your data warehouse](/warehouses) ### Troubleshooting Please see our [Google Tag Manager (GTM) Troubleshooting Guide](/kb/google-analytics/google-tag-manager-gtm-troubleshooting) and our [GA4 Troubleshooting Guide](/kb/google-analytics/google-analytics-ga4-troubleshooting) (if relevant) for help debugging GTM and GA4 integrations with GrowthBook. --- ## Migrate from LaunchDarkly import ExternalLink from '@site/src/components/ExternalLink' Looking to migrate from LaunchDarkly to GrowthBook? We have a dedicated importer tool that lets you get up and running in minutes. ## Create an API Token for LaunchDarkly GrowthBook uses the LaunchDarkly REST API to access the relevant resources and import them. You must create an access token in your [LaunchDarkly account authorization settings ](https://app.launchdarkly.com/settings/authorization). ## Run the Import Go to the [GrowthBook dashboard's Importing section ](https://app.growthbook.io/importing) and choose LaunchDarkly. Then, enter your LaunchDarkly access token and begin the 2-step process. ### Step 1: Fetch from LaunchDarkly This will fetch all projects, environments, and feature flags from LaunchDarkly. For large accounts with many feature flags, this may take several minutes to complete. If you run into rate limiting issues, you can adjust the default settings in the UI. When this completes, you will see a preview of everything that will be imported. Carefully review this before proceeding to step 2. Only items marked as "pending" will be imported when you proceed. :::note For feature flags, we do our best to accurately import everything from LaunchDarkly - fallback values, targeting rules, rollouts, and prerequisite features. This should cover most standard use cases, however for advanced set ups, there are likely some scenarios that we won't be able to handle properly. You can click the "open" link next to a feature flag to view a detailed JSON view of exactly what will be added/updated. Please review these carefully. If something doesn't look right, you can skip importing individual feature flags by clicking the "skip" link. ::: ### Step 2: Import to GrowthBook In this step, all of the "pending" resources will be imported into GrowthBook. This should take about 1 second per feature flag on average, although this can vary greatly. The import window must be left open during the entire process. If you are importing a feature that already exists in GrowthBook, a new revision will be created with any changes to the default value or rules. All of the changes will be published to all environments immediately, bypassing any draft/review process. ## Next Steps After the import is complete, you will need to replace the LaunchDarkly SDK in your application with the equivalent [GrowthBook SDK](/lib). --- ## How to Guides for installing the platform # GrowthBook Detailed Guides The following sections contain in-depth walkthroughs for setting up GrowthBook with various technologies. This list isn't complete! If you have a specific technology stack you'd like to see a guide for, please [let us know in Slack](https://growthbookusers.slack.com/archives/C01T6Q1SVFV). ## Tutorials ### Content Management Systems - [GrowthBook with Sanity](/guide/sanity) - [GrowthBook with Strapi](/guide/strapi) - [GrowthBook with Contentful](/guide/contentful) - [GrowthBook with Wordpress](/integrations/wordpress) ### No-Code/Low-Code Platforms - [GrowthBook with Google Tag Manager (GTM)](/guide/google-tag-manager-and-growthbook) - [GrowthBook with Shopify](/integrations/shopify) - [GrowthBook with Webflow](/integrations/webflow) ### JavaScript Frameworks - [GrowthBook with Deno and Hono](/guide/deno-hono) - [GrowthBook with Express.js](/guide/express-js) - [GrowthBook with Next.js, Vercel Edge Config, and Vercel Feature Flags API](/guide/nextjs-and-vercel-feature-flags) - [GrowthBook with Next.js (App Router)](/guide/nextjs-app-router) - [GrowthBook with Next.js (Pages Router)](/guide/nextjs-and-growthbook) - [GrowthBook with Create React App](/guide/create-react-app-and-growthbook) - [GrowthBook with Next.js and Rudderstack](/guide/rudderstack-and-nextjs-with-growthbook) ### Migration Guides - [How to Migrate from Statsig](/guide/migrate-from-statsig) - [How to Migrate from LaunchDarkly](/guide/importing) ## A/B Testing Guide - [The Open Guide to Successful A/B Testing (PDF)](/open-guide-to-ab-testing.v1.0.pdf) --- ## Migrate from Statsig to GrowthBook import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Migrate from Statsig This guide walks through migrating feature flags, experiments, and SDKs from Statsig to GrowthBook. ## Import Your Statsig Data Use GrowthBook's import tool to migrate your Statsig configuration and experiments. ![GrowthBook Statsig importer](/images/guides/import-from-statsig-with-data.png) ### Step 1: Generate a Statsig Console API Key - In Statsig, go to **Settings** → **Keys & Environments** - Create a new "Console" key with read-only permissions - Copy the key (it starts with `console-`; server `secret-` and client `client-` keys will not work) ### Step 2: Run the Import - In GrowthBook, go to **Settings** → **Import your data** - Select Statsig - Enter your Statsig Console API key - Choose the GrowthBook project to import into, and select a Data Source if you want to import Metrics - Click **Fetch from Statsig** The import tool will retrieve: - Environments (production, staging, etc.) - Tags - Segments - Feature Gates - Dynamic Configs - Experiments - Metric Sources (beta) - Metrics (beta) :::note A few things do not come over automatically yet: - Metrics and Metric Sources require a connected Data Source at import time, and only warehouse-native Statsig metrics are supported - Imported experiments do not include their metrics or exposure query — assign those in GrowthBook after importing ::: ### Step 3: Review and Confirm Items will show as: - **create**: New item that will be added to GrowthBook - **update**: Item already exists in GrowthBook and will be updated - **skip**: Item is unchecked and will be left untouched You can: - Uncheck items to remove them from the import - Filter items by tag, or show only new or updated items - Click the arrow (>) to view an item's Statsig definition and, for existing items, a diff against what's in GrowthBook - Click **Import to GrowthBook** when ready When the import status is marked as completed, your data is available in GrowthBook 🎉 ## Migrate Your SDKs **Automated SDK Migration** Use our AI-powered migration assistant to automatically refactor your code from Statsig to GrowthBook SDKs. **Supported SDKs** - JavaScript - Typescript - React (client-side) :::info Looking for another language? Reach out to support@growthbook.io. We're happy to help! ::: ### Installation Download the subagent to your Claude Code agents directory: ```bash # Create the agents directory if it doesn't exist mkdir -p ~/.claude/agents/ # Download the agent file curl -o ~/.claude/agents/statsig-to-growthbook-sdk-migrator.md \ https://raw.githubusercontent.com/growthbook/claude-statsig-to-growthbook-sdk-migrator/main/.claude/agents/statsig-to-growthbook-sdk-migrator.md ``` ### Using the Subagent In Claude Code, simply ask: > "I need to migrate this code from the Statsig SDK to the GrowthBook SDK: (@filename or directory)" The subagent will: - Convert import statements to GrowthBook SDK - Transform feature gates to boolean flags - Migrate dynamic configs to feature values - Flatten `StatsigUser` objects to GrowthBook attributes - Generate a migration report with any manual steps needed 🔗 See the [GitHub repo](https://github.com/growthbook/claude-statsig-to-growthbook-sdk-migrator) for more details. ## Terminology Mapping | Statsig | GrowthBook | | --------------- | ---------------------------------------------- | | Segments | Saved Groups | | Feature Gates | Features (boolean) | | Dynamic Configs | Features (JSON) + [Configs](/features/configs) | | Experiments | Experiments | | Metric Sources | Fact Tables | | Metrics | Fact Metrics | A Statsig Dynamic Config maps to a GrowthBook feature flag with a JSON value. To get the typed, schema-validated side of a Dynamic Config, back the flag with a [Config](/features/configs) (Enterprise) — a reusable object that carries a value and a schema, which is enforced on any flag it backs. The import tool creates plain JSON features; attaching a config is a manual step afterward. For simple reusable values shared across flags, see [Constants](/features/constants). ## Next Steps 1. Test your imported features in a development environment 2. Assign metrics and an exposure query to your imported experiments 3. Configure webhooks and integrations 4. Review team permissions and access controls Questions or issues? [Join our Slack community](https://join.slack.com/t/growthbookusers/shared_invite/zt-2xw8fu279-Y~hwnfCEf7WrEI9qScHURQ). --- ## GrowthBook and Next.js (Pages Router) import MaxWidthImage from "@site/src/components/MaxWidthImage"; # GrowthBook and Next.js (Pages Router) Note: If you are using the newer Next.js App Router, check out [our dedicated App Router guide](/guide/nextjs-app-router) instead. This document is a guide on how to add GrowthBook feature flags to your Next.js application. It assumes you are starting from scratch, so if you already have a Next.js application, you can skip to step 2. Here is a video version similar to the steps below: ## 1. Create your Next.js App Let's start by getting a basic Next.js app running: ```bash yarn create next-app ``` Then cd into the newly create directory and run: ```bash cd my-app yarn dev -p 4000 ``` Note: Both GrowthBook and Next.js run on port 3000 by default, so we're making our Next.js app use 4000 instead to avoid conflicts. Visit `http://localhost:4000` and you should see the application running! ## 2. GrowthBook Account You will need a GrowthBook account. You can either run GrowthBook locally or using the cloud hosted GrowthBook at [https://app.growthbook.io](https://app.growthbook.io). If you are using the GrowthBook cloud, you can skip to step 3. If you are installing it locally, here is the quick start instructions - or you can follow the [self hosting](/self-host) instructions. ```bash git clone https://github.com/growthbook/growthbook.git cd growthbook docker compose up -d ``` After that, visit `http://localhost:3000` and create your first user account. ![GrowthBook Signup Screen](/images/guides/nextjs-1-growthbook-signup-screen.jpeg) ## 3. Integrate the GrowthBook React SDK into our Next.js app GrowthBook will generate some integration code for you, including a unique SDK Client Key to load your features from. ![GrowthBook Integration Code](/images/guides/nextjs-2-feature-flag-key.png) Create the file `.env.local` if it doesn't exist yet and add your generated key there: ```bash NEXT_PUBLIC_GROWTHBOOK_API_HOST=https://cdn.growthbook.io NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY= # Below is only required if you enabled encryption NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY= ``` We first need to install the GrowthBook React SDK in our Next.js app: ```bash yarn add @growthbook/growthbook-react ``` Then we can modify the generated React code to work with the Next.js framework. Modify the file `pages/_app.js` with the following contents: ```ts import "../styles/globals.css"; import { GrowthBook, GrowthBookProvider } from "@growthbook/growthbook-react"; import { useEffect } from "react"; // Create a GrowthBook instance const growthbook = new GrowthBook({ apiHost: process.env.NEXT_PUBLIC_GROWTHBOOK_API_HOST, clientKey: process.env.NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY, decryptionKey: process.env.NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY, trackingCallback: (experiment, result) => { console.log("Viewed Experiment", { experimentId: experiment.key, variationId: result.key }); }, }); // Start downloading feature definitions growthbook.init(); export default function MyApp({ Component, pageProps, router }) { // Refresh features and targeting attributes on navigation useEffect(() => { growthbook.setURL(window.location.href); growthbook.setAttributes({ id: "123", loggedIn: true, deviceId: "abcdef123456", employee: true, company: "acme", country: "US", browser: navigator.userAgent, url: router.pathname, }); }, [router.pathname]); return ( ); } ``` In a real application, you would pull some of the targeting attributes from your authentication system or an API, but let's just leave them hard-coded for now. ## 4. Create a Feature in GrowthBook Back in the GrowthBook application, we can create a new feature. For this tutorial, we'll make a simple on/off feature flag that determines whether or not we show a welcome banner on our site. ![GrowthBook Create Feature](/images/guides/nextjs-3-create-feature.png) The key we chose (welcome-message) is what we will reference when using the GrowthBook SDK. We can now edit `pages/index.js` and conditionally render a welcome message based on the state of the feature: Add an import statement: ```ts import { IfFeatureEnabled } from "@growthbook/growthbook-react"; ``` And then put your welcome message somewhere on the page: ```ts I hope you enjoy this site and have a great day! ``` If you refresh your Next.js app, you'll notice the welcome message is not rendered. This is because when creating the feature, we set the default value to off. At this point, we could safely deploy our change to production and not worry about breaking anything. ## 5. Turn on the feature for your team Now we can add rules to the feature to turn it on for specific groups of users. In the hard-coded targeting attributes we set in pages/\_app.js, we designated ourselves as an internal employee. Let's use this attribute to turn on the feature for the whole internal team: ![GrowthBook Targeting Rule](/images/guides/nextjs-4-employee-rule.png) Refresh your Next.js app and you should now see the welcome message appearing! (Note: it might take up to 30 seconds for the API cache to refresh). ![Next.js app with feature](/images/guides/nextjs-5-nextjs-title.png) If you change employee to false in pages/\_app.js, you should see the welcome message disappear. The best part about targeting attributes in GrowthBook is that they are evaluated entirely locally. Sensitive user data is never sent over the network and there is no performance penalty. Some other libraries require an HTTP request to evaluate a feature for a user and this is often a deal breaker. ## 6. Gradually roll out to your users After you test the new feature within your team, you probably want to start rolling it out to real users. We can do that with another rule in GrowthBook: ![GrowthBook Rollout Rule](/images/guides/nextjs-6-rollout-rule.png) In the targeting attributes in pages/\_app.js, make sure employee is set to false. That will ensure you skip the first rule we made and fall into the second rollout rule. :::note The GrowthBook SDK uses deterministic hashing to figure out whether or not someone is included in a rollout (or A/B test). The SDKs hash together the selected targeting attribute (id) and the feature key (welcome-message) and coverts it to a float between 0 and 1. If that float is less than or equal to the rollout percent, the user is included. This ensures a consistent UX and prevents one user from constantly switching between ON and OFF as they navigate your app. ::: Try changing the user id in the targeting attributes in `pages/_app.js` to a few different random strings and see what happens. You should notice about half of the time the welcome message shows up and half of the time it doesn't. ## Conclusion and Next Steps We showed here how to do a targeted rule, and how to do a rollout rule. It's also just as easy to make an A/B test in the same manner. You will need to set up an event tracking and connect GrowthBook to your data source. You can look at the [GrowthBook React SDK](/lib/react) docs for more ways to use feature flags in your code besides the `` component. Once you do the initial integration work, it only takes a few seconds to wrap your code in feature flags. Once you realize how easy and stress-free deploys and experimentation can be, there's no going back. --- ## How to Integrate GrowthBook with Next.js, Vercel Edge Config, and Vercel Feature Flags import ExternalLink from '@site/src/components/ExternalLink' # Integrating GrowthBook with Next.js and Vercel Feature Flags This tutorial shows you how to use GrowthBook with Vercel's Edge Config and Feature Flags SDK to manage feature flags in a Next.js app. You'll need a Next app deployed on Vercel to follow along. While it's not necessary to use GrowthBook with Vercel's platform (see our other [guides on adding GrowthBook to Next.js](/guide/)), this setup provides several benefits: - **Performance**: [Vercel Edge Config](https://vercel.com/docs/storage/edge-config)  is a global data store that provides near-instantaneous access to your data, making it perfect for handling feature flags. - **Streamlined**: The GrowthBook SDK automatically sends feature flag values to Vercel Edge Config, keeping your data in sync. - **Developer Experience**: The Vercel Toolbar enables your team to inspect feature flags, locally override their values, and see changes. ## Use Vercel Edge Config to Store GrowthBook Feature Flags To set up Edge Config: 1. Log in to Vercel and select your project. Navigate to **Storage**. 2. Create a new **Edge Config** and save the **ID** (starting with `ecfg_`) to use later. If you're part of a team in Vercel, also save your Team ID. Note that even personal accounts may have a team. 3. Navigate to **Account Settings** → **Tokens** and create a Vercel API Token. Save the value to use in the next step. ## Create a GrowthBook SDK Webhook In GrowthBook: 1. Navigate to **SDK Connections**. Open or create a JavaScript SDK connection. 2. Add a new **SDK Webhook** and set the type to `Vercel Edge Config`. 3. Add your Edge Config ID, Vercel API Token, and Team ID. ![GrowthBook SDK Webhook](/images/guides/vercel-edge-sdk-connection.webp) After you add the webhook, click **Test** to verify it's working correctly. :::note If you receive an `items exceed size limit` error, it means your feature flag payload is too large. Vercel limits the [size of your Edge Config store](https://vercel.com/docs/storage/edge-config/edge-config-limits#limits-by-plan) depending on your plan. You can either reduce the number of flags or upgrade your Vercel plan. ::: ## Configure Next.js In your Next app, install the required dependencies: ```bash npm2yarn npm install @growthbook/growthbook @vercel/flags @vercel/edge-config ``` ### flags.ts Create a `utils/flags.ts` file in your project. This file configures the GrowthBook SDK and defines some helper functions to use feature flags throughout your app. ```typescript import { Attributes, GrowthBook, GrowthBookPayload, WidenPrimitives, } from "@growthbook/growthbook"; import { cookies } from "next/headers"; import { cache } from "react"; import { get } from "@vercel/edge-config"; import { unstable_flag as flag } from "@vercel/flags/next"; import { ApiData, JsonValue } from "@vercel/flags"; const GROWTHBOOK_DOMAIN = "https://app.growthbook.io"; const EDGE_CONFIG_KEY = "gb_payload"; // Get targeting attributes from cookies/headers function getAttributes(): Attributes { const cookieStore = cookies(); return { id: cookieStore.get("gbuuid")?.value, }; } /* **** Do not edit below this line **** */ // Get feature definitions from Edge Config async function getPayload(): Promise { try { const payload: GrowthBookPayload | undefined = await get(EDGE_CONFIG_KEY); if (typeof payload !== "string") { console.log("Invalid GrowthBook payload", payload); return {}; } return JSON.parse(payload); } catch (e) { console.log("Error parsing GrowthBook payload", e); } return {}; // Return empty payload as a fallback } // Return a memoized GrowthBook instance for a request const getGrowthBookInstance = cache(async () => { return new GrowthBook({ attributes: getAttributes() }).initSync({ payload: await getPayload(), }); }); // Generate feature flag definitions for the Vercel Toolbar export async function getFlagApiData(): Promise { const payload = await getPayload(); const data: ApiData = {}; Object.entries(payload.features || {}).forEach(([key, value]) => { const options = new Set([value.defaultValue]); // For boolean flags, make sure true/false are always added as options if (value.defaultValue === false) options.add(true); if (value.defaultValue === true) options.add(false); // Loop through rules and add all the possible values (value.rules || []).forEach((rule) => { // Force and rollout rules if (rule.force) { options.add(rule.force); } // Experiment rules if (rule.variations) { rule.variations.forEach((variation) => options.add(variation)); } }); data.definitions = data.definitions || {}; data.definitions[key] = { origin: `${GROWTHBOOK_DOMAIN}/features/${key}`, options: Array.from(options).map((value) => ({ value })), }; }); return data; } // Helper function to get tracking data for experiments export interface FlagTrackData { feature: string; experimentId: string; variationId: string; hashAttribute: string; hashValue: string; } export async function getTrackData(): Promise { const gb = await getGrowthBookInstance(); const calls = gb.getDeferredTrackingCalls(); return calls.map((call) => ({ feature: call.result.featureId ?? "", experimentId: call.experiment.key, variationId: call.result.key, hashAttribute: call.result.hashAttribute, hashValue: call.result.hashValue, })); } // Helper function to evaluate a feature flag export function getFlagValue( key: string, defaultValue: T ) { return flag>({ key, defaultValue, options: [], decide: async () => { const gb = await getGrowthBookInstance(); return gb.getFeatureValue(key, defaultValue); }, })(); } ``` ### TrackFlags.tsx Create a `utils/TrackFlags.ts` file in your project. This client component sends a tracking event whenever a user is exposed to an A/B test. Replace the `console.log` with your own tracking callback. ```typescript "use client"; import { useEffect } from "react"; import { FlagTrackData } from "./flags"; // Only fire tracking calls once per page load const trackedExperiments = new Set(); export function TrackFlags({ data }: { data: FlagTrackData[] }) { useEffect(() => { data.forEach((info) => { const props = { experimentId: info.experimentId, variationId: info.variationId, [info.hashAttribute]: info.hashValue, }; const key = JSON.stringify(props); if (trackedExperiments.has(key)) return; trackedExperiments.add(key); // TODO: Send tracking call to analytics (GA4, Segment, etc.) console.log(`Experiment Viewed`, props); }); }); return null; } ``` ### Set Up Environment Variables Pull in environment variables from your Vercel project: ```bash npx vercel env pull ``` This command creates an `.env.local` file in your project with the `EDGE_CONFIG` key. ## Create Feature Flags in GrowthBook In GrowthBook, create 2 new feature flags: | Key | Type | Default Value | | ------------ | ------- | ------------- | | `showBanner` | boolean | `true` | | `headerText` | string | `My Header` | Your flags should look like this: ![Feature flags configured in GrowthBook](/images/guides/vercel-ff-features.webp) These simple flags demonstrate everything you need to know to create more complex feature flags with targeting rules, percent rollouts, and A/B tests. ## Use Feature Flags in Your Next.js App Open `page.tsx` (or your main component) and add: ```tsx import { getFlagValue, getTrackData } from "@/utils/flags"; import { TrackFlags } from "@/utils/TrackFlags"; export default async function Home() { // If the app can't connect to GrowthBook, it will use the default (second) values const showBanner = await getFlagValue("showBanner", false); const headerText = await getFlagValue("headerText", "fallback"); // If any A/B tests were run as part of the above flags, we want to track them client-side const trackData = await getTrackData(); return ( {showBanner && A Banner!} {headerText} ) } ``` Start your development server: ```bash npm2yarn npm run dev ``` Now you can see your feature flags in action! ## 🎉 Bonus: The Vercel Toolbar Use the Vercel Toolbar to locally override flag values and see real-time changes. ![Vercel Toolbar](/images/guides/vercel-toolbar.webp) ### Enable the Toolbar Create a new file, `app/.well-known/vercel/flags/route.ts`, in your project: ```ts import { type NextRequest, NextResponse } from "next/server"; import { ApiData, verifyAccess } from "@vercel/flags"; import { getFlagApiData } from "@/utils/flags"; export async function GET(request: NextRequest) { const access = await verifyAccess(request.headers.get("Authorization")); if (!access) return NextResponse.json(null, { status: 401 }); const data = await getFlagApiData(); return NextResponse.json(data); } ``` ### Set Up the `FLAGS_SECRET` Environment Variable 1. Generate a new secret key: ```bash node -e "console.log(crypto.randomBytes(32).toString('base64url'))" ``` 2. In your Vercel project settings, create a new environment variable called `FLAGS_SECRET`. Paste the generated key as the value. 3. Locally, run `npx vercel env pull` to update your `.env.local` file with the new environment variable. See [Vercel's docs](https://vercel.com/docs/workflow-collaboration/feature-flags/supporting-feature-flags#flags_secret-environment-variable) for more info. ### Refresh and Explore With everything in place, it's time to see the Vercel Toolbar in action. Refresh your app and click the toggle icon to see feature flag values and override them. You've got the power! ## Conclusion You've successfully integrated GrowthBook with Next.js and Vercel Feature Flags, giving you fine-grained control over your features, improved performance with Edge Config, and a streamlined development workflow. Now get out there and start experimenting 🧪 --- ## GrowthBook and Next.js (App Router) import MaxWidthImage from "@site/src/components/MaxWidthImage"; # GrowthBook and Next.js (App Router) Note: If you are using the older Next.js Pages Router, check out [our dedicated Pages Router guide](/guide/nextjs-and-growthbook) instead. This document is a guide on how to add GrowthBook feature flags to your Next.js application. It assumes you are starting from scratch, so if you already have a Next.js application, you can skip to step 2. ## 1. Create your Next.js App Let's start by getting a basic Next.js app running: ```bash yarn create next-app ``` Then cd into the newly create directory and run: ```bash cd my-app yarn dev -p 4000 ``` Note: Both GrowthBook and Next.js run on port 3000 by default, so we're making our Next.js app use 4000 instead to avoid conflicts. Visit `http://localhost:4000` and you should see the application running! ## 2. GrowthBook Account You will need a GrowthBook account. You can either run GrowthBook locally or using the cloud hosted GrowthBook at [https://app.growthbook.io](https://app.growthbook.io). If you are using the GrowthBook cloud, you can skip to step 3. If you are installing it locally, here is the quick start instructions - or you can follow the [self hosting](/self-host) instructions. ```bash git clone https://github.com/growthbook/growthbook.git cd growthbook docker compose up -d ``` After that, visit `http://localhost:3000` and create your first user account. ![GrowthBook Signup Screen](/images/guides/nextjs-1-growthbook-signup-screen.jpeg) ## 3. Set Environment Variables in your Next.js App In GrowthBook, create a new SDK Connection. After doing this, you should see an API Host and Client Key (and a Decryption Key if you enabled encryption). Create the file `.env.local` in your Next.js app if it doesn't exist yet and add this info there: ```bash NEXT_PUBLIC_GROWTHBOOK_API_HOST=https://cdn.growthbook.io NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY= # Below is only required if you enabled encryption NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY= ``` ## 4. Create a Feature Flag Back in the GrowthBook application, let's create our first feature flag. Make a simple ON/OFF feature flag with the key `welcome-message`. Leave the default value set to OFF. We will turn it on in a future step. ![GrowthBook Create Feature](/images/guides/nextjs-3-create-feature.png) ## 5. Integrate the GrowthBook SDK into our Next.js app Next.js App Router uses Server Components by default, so we need to use the GrowthBook JavaScript SDK, not the React SDK (which only works client-side). ```bash npm2yarn npm install --save @growthbook/growthbook ``` Our Javascript SDK works out-of-the-box with React Server Components, but we can more deeply integrate it with Next.js by creating a small helper function in `app/growthbookServer.ts` with the following contents: ```ts // app/growthbookServer.ts import { setPolyfills, configureCache } from "@growthbook/growthbook"; export function configureServerSideGrowthBook() { // Tag fetch requests so they can be revalidated on demand setPolyfills({ fetch: (url: string, init: RequestInit) => fetch(url, { ...init, next: { // Cache feature definitions for 10 seconds for dev // In prod, use a higher value and use WebHooks to revalidate on-demand revalidate: 10, tags: ["growthbook"], }, }), }); // Disable the built-in cache since we're using Next.js's fetch cache instead configureCache({ disableCache: true, }); } ``` Now, let's modify our main `app/page.tsx` file. Change the top of the file to match the following: ```tsx import Image from "next/image"; import { configureServerSideGrowthBook } from "./growthbookServer"; import { GrowthBook } from "@growthbook/growthbook"; export default async function Home() { configureServerSideGrowthBook(); // Create and initialize a GrowthBook instance const gb = new GrowthBook({ apiHost: process.env.NEXT_PUBLIC_GROWTHBOOK_API_HOST, clientKey: process.env.NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY, decryptionKey: process.env.NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY, }); await gb.init({ timeout: 1000 }); // Set targeting attributes for the user // TODO: get from cookies/headers/db await gb.setAttributes({ id: "123", employee: true, }); // Evaluate a feature flag const welcomeMessage = gb.isOn("welcome-message"); // Cleanup gb.destroy(); ``` Next Steps And now let's use the feature flag we referenced above. Add the following somewhere on the page so we can see it: ```tsx Welcome Message: {welcomeMessage ? "ON" : "OFF"} ``` This will render as `OFF` for now. ## 6. Turn on for your Team In our page, we hard-coded `employee: true` when setting our targeting attributes. Let's use that to turn on the feature for just employees. Create a Force Rule for your feature: ![GrowthBook Targeting Rule](/images/guides/nextjs-4-employee-rule.png) Publish the draft and refresh your Next.js app and you should now see the welcome message showing up as ON! (Note: it might take a few seconds for the cache to refresh). If you change your targeting attribute to `employee: false` in the page, you should see the welcome message switch back to OFF immediately. The best part about targeting attributes in GrowthBook is that they are evaluated entirely locally. Sensitive user data is never sent over the network and there is no performance penalty. Some other libraries require an HTTP request to evaluate a feature for a user and this is often a deal breaker for performance. ## 7. Gradually roll out to your users After you test the new feature within your team, you probably want to start rolling it out to real users. We can do that with another rule in GrowthBook: ![GrowthBook Rollout Rule](/images/guides/nextjs-6-rollout-rule.png) In the targeting attributes, make sure you set `employee: false`. That will ensure you skip the first rule we made and fall into the second rollout rule. :::note The GrowthBook SDK uses deterministic hashing to figure out whether or not someone is included in a rollout (or A/B test). The SDKs hash together the selected targeting attribute (id) and the feature key (welcome-message) and coverts it to a float between 0 and 1. If that float is less than or equal to the rollout percent, the user is included. This ensures a consistent UX and prevents one user from constantly switching between ON and OFF as they navigate your app. ::: Try changing the id in the targeting attributes to a few different random strings and see what happens. You should notice about half of the time the welcome message will be ON. ## Conclusion and Next Steps We showed here how to do a targeted rule, and how to do a rollout rule. It's also just as easy to make an A/B test in the same manner. You will need to set up event tracking and connect GrowthBook to your data source. We support many different rendering strategies in Next.js, not just server components. Check out our [example Next.js app](https://github.com/growthbook/examples/tree/main/next-js), which demonstrates static rendering, client components, streaming server components, and more. Once you do the initial integration work, it only takes a few seconds to wrap your code in feature flags. Once you realize how easy and stress-free deploys and experimentation can be, there's no going back. --- ## GrowthBook, Rudderstack, and Next.js import MaxWidthImage from "@site/src/components/MaxWidthImage"; # A/B Testing with Rudderstack and Next.js This document is a guide on how to add GrowthBook feature flags and A/B testing to your existing Next.js application using Rudderstack for event tracking. ## 1. Create a GrowthBook Account You will need a GrowthBook account. You can either run GrowthBook locally or using the cloud hosted GrowthBook at [https://app.growthbook.io](https://app.growthbook.io). If you are installing it locally, you can follow the self-hosting quick start instructions here: [self hosting instructions](/self-host). ## 2. Create a JS source in Rudderstack From within your Rudderstack account, create a new JS source. ![Add Rudderstack JS source](/images/guides/rudderstack-7-create-source.png) Name the source whatever you like, in this example I'm using `GrowthBook JS`. When the source is created, connect it to your BigQuery data warehouse (or whatever destination you're using for GrowthBook experiment data). You can read more about how to connect to your data destination [here](/guide/rudderstack). ![Add BigQuery destination](/images/guides/rudderstack-8-source-w-destination.png) Once you have it connected, copy the write key, as we'll need it for the next step. Under connections, you should now see the JS source connected to the BigQuery destination. You will also need the `Data plane URL` which appears near the top of the page. ![Rudderstack BigQuery connection](/images/guides/rudderstack-9-connected-js.png) ## 3. Integrate Rudderstack into your Next.js application While there is plenty of documentation on how to add Rudderstack to your Next.js application out there, none of those implementations are very Next.js like, and limit the ability of Rudderstack to integrate more deeply into your code- including using GrowthBook. Below is the integration code that we came up with to address these concerns. ### install the Rudderstack Analytics package Install the javascript SDK for Rudderstack with yarn, ```bash yarn add rudder-sdk-js ``` or npm: ```bash npm install --save rudder-sdk-js ``` ### Create Rudderstack loader Create a `rudder.js` file in your Next.js project. This file will load Rudderstack's SDK in a reusable and asynchronous way. ```javascript let rudder; async function getInstance() { if (!rudder) { rudder = await import("rudder-sdk-js"); rudder.load( process.env.NEXT_PUBLIC_RUDDERSTACK_KEY, process.env.NEXT_PUBLIC_RUDDERSTACK_HOST, { integrations: { All: true } } ); await new Promise((resolve) => rudder.ready(resolve)); } return rudder; } const rudderObj = { init: getInstance, track: (...args) => getInstance().then((r) => r.track(...args)), getAnonymousId: async () => getInstance().then((r) => r.getAnonymousId()), }; export default rudderObj; ``` If you want to add other methods, like `identify`, you can extend the rudderObj. You'll also have to add the Rudderstack Key and Host to your environment variables, or add to your `.env.local` file: ```javascript NEXT_PUBLIC_RUDDERSTACK_KEY= NEXT_PUBLIC_RUDDERSTACK_HOST=https:// ``` The key is the `write key` from the JS source we made in step 2, and the HOST is the `data plane URL`. ### Integrate Rudderstack into your Next.js application In your `_app.js`, add the Rudderstack integration we just created ```javascript import rudder from "./rudder"; ``` This will allow you add `rudder.track()` in your app anywhere you import rudder.js while sharing the same Rudderstack object. ## 4. Integrate the GrowthBook React SDK into our Next.js app We first need to install the GrowthBook React SDK in our Next.js app: ```bash yarn add @growthbook/growthbook-react ``` Get the API key from GrowthBook (under settings → API or from the top of the implementation instructions) and add to your environment variables (.env.local) ```js NEXT_PUBLIC_GROWTHBOOK_FEATURES_URL= ``` Then we can modify the code to work with GrowthBook. Modify the file `pages/_app.js` to add GrowthBook and Rudderstack. Import GrowthBook (and Rudderstack, if you haven't): ```ts import { GrowthBook, GrowthBookProvider, useFeature, } from "@growthbook/growthbook-react"; import rudder from "./rudder"; ``` then create your GrowthBook instance: ```ts // Create a GrowthBook instance const growthbook = new GrowthBook({ trackingCallback: (experiment, result) => { rudder.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.variationId, }); }, }); ``` :::note The names `experiment Viewed`, `experimentId` and `variationId` will be mapped to `experiment_id` and `variation_id` columns in the `experiment_viewed` table within BigQuery ::: Then add a `useEffect` hook to update Rudderstack and GrowthBook when the page changes. ```ts export default function MyApp({ Component, pageProps }) { useEffect(() => { // Load feature definitions from API fetch(process.env.NEXT_PUBLIC_GROWTHBOOK_FEATURES_URL) .then((res) => res.json()) .then((json) => { growthbook.setFeatures(json.features); }); // TODO: replace with real targeting attributes growthbook.setAttributes({ company: "foo", browser: "foo", url: "foo", }); // Add in Rudderstack anonId when loaded rudder.getAnonymousId().then((id) => { growthbook.setAttributes({ ...growthbook.getAttributes(), id }); }); }, []); //... } ``` This code adds the `id` with in GrowthBook to the Rudderstack anonymous_id. If you want to load user_id as well as anonymous_id you'll have to add this id to the setAttribute, and also call the `rudder.identify()` with the user_id info. Finally, wrap your Next.js project in the GrowthBookProvider component, so we can use the GrowthBook methods throughout the codebase without doing addition instantiation. ```ts return ( ); ``` All together, your `_app.js` should look something like this: ```ts import "../styles/globals.css"; import { GrowthBook, GrowthBookProvider, useFeature, } from "@growthbook/growthbook-react"; import { useEffect } from "react"; import rudder from "./rudder"; // Create a GrowthBook instance const growthbook = new GrowthBook({ trackingCallback: (experiment, result) => { rudder.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.variationId, }); }, }); export default function MyApp({ Component, pageProps }) { useEffect(() => { // Load feature definitions from API fetch(process.env.NEXT_PUBLIC_GROWTHBOOK_FEATURES_URL) .then((res) => res.json()) .then((json) => { growthbook.setFeatures(json.features); }); // TODO: replace with real targeting attributes growthbook.setAttributes({ company: "foo", browser: "foo", url: "foo", }); // Add in Rudderstack anonId when loaded rudder.getAnonymousId().then((id) => { growthbook.setAttributes({ ...growthbook.getAttributes(), id }); }); }, []); return ( ); } ``` Once you have data flowing into Rudderstack, you can set it up to work with GrowthBook by following the [Rudderstack guide](/guide/rudderstack) --- ## Sanity with GrowthBook import MaxWidthImage from '@site/src/components/MaxWidthImage'; This comprehensive tutorial will walk you through setting up Sanity with GrowthBook for A/B testing in a Next.js application. You'll learn how to create content variations with Sanity and use GrowthBook for feature flag experiments like in the example below. Sanity is a real-time content backend for all your text and assets. Sanity Studio is a flexible, multiplayer TypeScript CMS which is configured with code and deployed to your authors. In this guide you’ll embed it as part of your Next.js application, but it can also be deployed standalone. GrowthBook (that's us!) is open source feature flag management and experimentation platform that works great with your stack—including Sanity. For the purposes of this guide, we'll be creating an A/B test for the product page of a fictional camera store, Camera World. ![Two variations of Camera World with different titles and images](/images/guides/sanity-ab.png) ## Prerequisites - Node.js 20+ installed - A [GrowthBook account](https://www.growthbook.io) - A [Sanity account](https://www.sanity.io) - Basic knowledge of Next.js and React ## Video Tutorial ## Set up Next.js and Sanity ### Install Next.js Already have a Next.js project? Skip to the next section. ```bash npm2yarn npx create-next-app@latest my-sanity-growthbook-app --typescript --tailwind --eslint --app ``` ### Install Sanity Inside your Next.js project, install Sanity: ```bash npm2yarn npx sanity@latest init ``` Choose the default options when prompted. This creates Sanity Studio inside your Next.js project (accessible at `/studio`) and sets up the necessary config files. :::info Free Sanity account required You’ll need a free Sanity account to complete this guide, if you do not yet have one, you will be prompted to create one after running this command. ::: ## Add the Sanity Personalization Plugin Sanity offers a plugin for personalization that includes a [direct integration with GrowthBook](https://github.com/sanity-io/sanity-plugin-personalization/blob/main/growthbook.md). This plugin allows you to create content variations within Sanity and use GrowthBook for feature flag experiments. Install the plugin: ```bash npm2yarn npm install @sanity/personalization-plugin ``` ### Update your Sanity config In your Next.js project, update `sanity.config.ts` to include the personalization plugin. Note that the `fields` listed below are the existing defined fields that you are extending to be able to experiment on. (Changes highlighted in the code block below.) ```ts "use client"; /** * This configuration is used to for the Sanity Studio that’s mounted on the `/app/studio/[[...tool]]/page.tsx` route */ import { visionTool } from "@sanity/vision"; import { defineConfig } from "sanity"; import { structureTool } from "sanity/structure"; // highlight-next-line import { fieldLevelExperiments } from "@sanity/personalization-plugin/growthbook"; // Go to https://www.sanity.io/docs/api-versioning to learn how API versioning works import { apiVersion, dataset, projectId } from "./sanity/env"; import { schema } from "./sanity/schemaTypes"; import { structure } from "./sanity/structure"; export default defineConfig({ basePath: "/studio", projectId, dataset, // Add and edit the content schema in the './sanity/schemaTypes' folder schema, plugins: [ structureTool({ structure }), // Vision is for querying with GROQ from inside the Studio // https://www.sanity.io/docs/the-vision-plugin visionTool({ defaultApiVersion: apiVersion }), // highlight-start fieldLevelExperiments({ fields: ["string", "image"], environment: "", // Change to your GrowthBook environment project: "", // Optional: specify your GrowthBook project ID }), // highlight-end ], }); ``` ### Add Experiment Fields to Your Sanity Schema With the personalization plugin installed, you can now add experiment fields to your Sanity schema. Create a new file `sanity/schemaTypes/camera.ts`: ```ts import { defineField, defineType } from "sanity"; export const camera = defineType({ name: "camera", title: "Camera Product", type: "document", fields: [ defineField({ name: "slug", type: "slug", options: { source: "title.default", }, }), defineField({ name: "title", type: "experimentString", }), defineField({ name: "price", type: "number", }), defineField({ name: "image", type: "experimentImage", }), defineField({ name: "description", type: "experimentString", }), defineField({ name: "specs", title: "Specifications", type: "object", fields: [ { name: "megapixels", title: "Megapixels", type: "string" }, { name: "sensor", title: "Sensor Type", type: "string" }, { name: "iso", title: "ISO Range", type: "string" }, { name: "videoResolution", title: "Video Resolution", type: "string" }, { name: "battery", title: "Battery Life", type: "string" }, { name: "weight", title: "Weight", type: "string" }, { name: "connectivity", title: "Connectivity", type: "string" }, ], }), defineField({ name: "features", title: "Key Features", type: "array", of: [{ type: "string" }], options: { layout: "tags" }, }), defineField({ name: "inStock", type: "boolean", initialValue: true, }), defineField({ name: "category", type: "string", options: { list: [ { title: "DSLR", value: "dslr" }, { title: "Mirrorless", value: "mirrorless" }, { title: "Point & Shoot", value: "pointshoot" }, { title: "Action Camera", value: "action" }, { title: "Instant Camera", value: "instant" }, ], }, }), ], preview: { select: { title: "title.default", subtitle: "category", media: "image.default", }, }, }); ``` The schema uses `experimentString` and `experimentImage` field types (highlighted in the code block above) provided by the personalization plugin. These allow you to create content variations with Sanity and use GrowthBook for experiments. Import the schema into your `sanity/schemaTypes.ts` file: ```ts import { type SchemaTypeDefinition } from "sanity"; import { camera } from "./camera"; export const schema: { types: SchemaTypeDefinition[] } = { types: [camera], }; ``` ### Add Queries to Fetch Experiment Content Create a new file `sanity/lib/queries.ts` to store your GROQ query: ```ts import { defineQuery } from "next-sanity"; export const CAMERA_PRODUCT_QUERY = defineQuery(` *[_type == "camera" && slug.current == $slug][0]{ "title": coalesce( title.variants[experimentId == $titleExperiment && variantId == $titleVariant][0].value, title.default ), "image": coalesce( image.variants[experimentId == $imageExperiment && variantId == $imageVariant][0].value, image.default ), slug, "description": coalesce( description.variants[experimentId == $descriptionExperiment && variantId == $descriptionVariant][0].value, description.default ), specs, features, inStock, category, price } `); ``` This query fetches the experiment content and serves the appropriate variation to the user. :::tip Type generation Run `npx sanity@latest schema extract && npx sanity@latest typegen generate` in your project to generate types for your queries/schema automatically. ::: ## Configure GrowthBook In this section, we'll create an SDK connection, feature flag, and experiment in GrowthBook. - The SDK connection links GrowthBook to your Next.js app. - Feature flags live in your codebase and control which code is executed. - Experiments are defined as feature flag rules and are used to test different variations. ### Create a GrowthBook SDK Connection ![GrowthBook SDK Connection](/images/guides/sanity-sdk-connection.png) 1. In GrowthBook, go to **SDK Configuration** → **SDK Connections** and click **Add SDK Connection**. 2. Choose **Node.js** and click **Save**. 3. Copy the **Client Key** and add it to your `.env.local` in Next.js. (You'll also see the environment variables for Sanity in the file.) ```env GROWTHBOOK_CLIENT_KEY= ``` :::tip Update env variables Remember to update environment variables with your hosting provider when you deploy your app. ::: ### Create a Feature Flag For our Camera World example, let's test the title, description, and image for the Sony A7 IV product page. ![GrowthBook Feature Flag](/images/guides/sanity-feature-flag.png) 1. In GrowthBook, go to **Features** and click **Add Feature**. 2. Give the feature a name, like `flag-sony-a7-iv`. 3. Set the value type to **String**. 4. Set the default value to "control". 5. Save the feature. ### Create an Experiment With the feature flag created, we can now create an experiment. ![GrowthBook Experiment](/images/guides/sanity-experiment.png) 1. On the same page, scroll down to **Rules** and click **Add Rule**. 2. Select **Experiment**. 3. Configure your experiment: - Name: "Sony A7 IV Experiment" - Control: "control" - Variation 1: "variation" 4. Complete remaining steps and save. 5. Click **Review & Publish** to launch the experiment. Your experiment is now live! Before creating the content variations with Sanity, we'll need to create an API Key that allows Sanity to fetch the experiment content. ### Create an API Key 1. In GrowthBook, go to **Settings** → **API Keys** 2. Click **New Secret Key** 3. Add a description, like `Sanity API Key`. 4. Set the role to **readonly**. 5. Save and copy the API key for the next step. ## Add Content Variations with Sanity In Sanity Studio, we'll open the `Camera Product` document for the Sony A7 IV. While our control uses matter-of-fact language, the variation will be more exciting. :::note API key required The first time you open document an experiment field, you'll be prompted to enter the API key you created in the previous step. ::: 1. Add your default content. 2. Hover over the **Title**, **Description**, and **Image** fields to reveal an experiment (flask) icon. 3. Click the experiment icon to open a dropdown where you can select the experiment you created in GrowthBook. 4. Add content for each variation: - **Control**: Your default content - **Variation**: Alternative title/image to test 5. Publish content! Here's what the experiment fields look like in Sanity Studio: ![Sanity Experiment Field](/images/guides/sanity-add-experiment-field.png) ![Add flag to field](/images/guides/sanity-add-flag-to-field.png) ![Sanity experiment](/images/guides/sanity-experiment-set.png) ## Add GrowthBook to Your Next.js App Now, let's integrate GrowthBook in your Next.js app to serve the appropriate content to the user. ### Install GrowthBook ```bash npm2yarn npm install @growthbook/growthbook ``` ### Configure GrowthBook for Server-Side Rendering Create a new file, `lib/growthbookServer.ts`: ```ts import { setPolyfills, configureCache } from "@growthbook/growthbook"; export function configureServerSideGrowthBook() { // Tag fetch requests so they can be revalidated on demand setPolyfills({ fetch: ( url: Parameters[0], opts: Parameters[1] ) => fetch(url, { ...opts, next: { // Cache feature definitions for 1 minute // Implement SDK webhooks to revalidate on demand (see gb-revalidate route handler) revalidate: 60, tags: ["growthbook"], }, }), }); // Disable the built-in cache since we're using Next.js's fetch cache instead configureCache({ disableCache: true, }); } ``` ### Configure Experiment Tracking To track experiment exposures—which variations a users saw—we'll need to create a tracking component. Create a new file, `lib/growthbookTracking.tsx`. Note the `TODO` comment. You'll need to replace the `console.log` with your own [event tracking](/app/datasources). Popular event tracking tools include Google Analytics, Segment, and Jitsu. ```tsx "use client"; import type { TrackingCallback, TrackingData } from "@growthbook/growthbook"; import { useEffect } from "react"; export const onExperimentView: TrackingCallback = (experiment, result) => { // TODO: Replace with your analytics service (Google Analytics, Segment, etc.) console.log("Viewed Experiment", { experimentId: experiment.key, variationId: result.key, }); // Example: Google Analytics 4 // if (typeof window !== 'undefined' && window.gtag) { // window.gtag('event', 'experiment_view', { // experiment_id: experiment.key, // variant_id: result.key, // }); // } }; // Helper component to track experiment views from server components export function GrowthBookTracking({ data }: { data: TrackingData[] }) { useEffect(() => { data.forEach(({ experiment, result }) => { onExperimentView(experiment, result); }); }, [data]); return null; } ``` ### Update Middleware for Consistent User IDs Create or update `middleware.ts` in your project root to ensure consistent experiment assignment. ```ts import { NextResponse } from "next/server"; import type { NextRequest } from "next/server"; // Ensure a UUID cookie is always set to enable experimentation export const GB_UUID_COOKIE = "gb-user-id"; export async function middleware(request: NextRequest) { // Generate a UUID if it doesn't exist yet let uuid = request.cookies.get(GB_UUID_COOKIE)?.value; let needsUpdate = false; if (!uuid) { uuid = crypto.randomUUID(); needsUpdate = true; // Set the cookie on the current request request.cookies.set(GB_UUID_COOKIE, uuid); } // Forward the request to the server const response = NextResponse.next({ request: { headers: new Headers(request.headers), }, }); // Add the newly created UUID to the response headers to persist in the browser if (needsUpdate) { response.cookies.set(GB_UUID_COOKIE, uuid); } return response; } // Run for all pages that need experiments export const config = { matcher: [ '/((?!api|_next/static|_next/image|favicon.ico|studio).*)', ], }; ``` ## Bring Everything Together in Your Next.js App Now let's create a page that demonstrates the A/B testing functionality. It's a dynamic route that fetches the product data from Sanity and serves the appropriate content to the user. The relevant part of the template is highlighted in the code block below. ![Sanity experiment: a](/images/guides/sanity-a.png) ![Sanity experiment: b](/images/guides/sanity-b.png) ### Create an Experiment Page Edit `app/[slug]/page.tsx`: ```tsx // app/[slug]/page.tsx import { GrowthBookTracking } from "@/lib/GrowthBookTracking"; import { configureServerSideGrowthBook } from "@/lib/growthbookServer"; import { sanityFetch } from "@/sanity/lib/live"; import { CAMERA_PRODUCT_QUERY } from "@/sanity/lib/queries"; import { urlFor } from "@/sanity/lib/sanityImageUrl"; import { GrowthBook } from "@growthbook/growthbook"; import { cookies } from "next/headers"; import { GB_UUID_COOKIE } from "@/middleware"; import Image from "next/image"; import { CAMERA_PRODUCT_QUERYResult } from "@/sanity.types"; type Props = { params: Promise; }; const EXPERIMENT_FLAG = "flag-sony-a7-iv"; export default async function ProductPage({ params }: Props) { // Helper to configure cache for next.js configureServerSideGrowthBook(); // Create and initialize a GrowthBook instance const gb = new GrowthBook({ clientKey: process.env.GROWTHBOOK_CLIENT_KEY, enableDevMode: true, }); await gb.init({ timeout: 1000 }); // Set targeting attributes for the user const cookieStore = await cookies(); await gb.setAttributes({ id: cookieStore.get(GB_UUID_COOKIE)?.value || "", }); // Evaluate any feature flags const variation = gb.getFeatureValue("flag-sony-a7-iv", "default"); // Get tracking data for experiments const trackingData = gb.getDeferredTrackingCalls(); const queryParams = { slug: (await params).slug, titleExperiment: EXPERIMENT_FLAG, titleVariant: variation, descriptionExperiment: EXPERIMENT_FLAG, descriptionVariant: variation, imageExperiment: EXPERIMENT_FLAG, imageVariant: variation, }; const { data } = await sanityFetch({ query: CAMERA_PRODUCT_QUERY, params: queryParams, }) as { data: CAMERA_PRODUCT_QUERYResult }; // Cleanup gb.destroy(); if (!data) { return ( Product Not Found The camera you're looking for doesn't exist. ); } return ( {/* Header */} 📸 Camera World Cameras Lenses Accessories Support {/* Product Content */} {/* Product Image */} //highlight-start {data.image ? ( ) : ( 📷 )} //highlight-end {/* A/B Test Info (for demo purposes) */} 🧪 A/B Test Status: {EXPERIMENT_FLAG} Title Variant:{" "} {data.title} ({variation === "variation" ? "🅱️" : "🅰️"}) Description Variant:{" "} {data.description} ({variation === "variation" ? "🅱️" : "🅰️"}) Image Variant:{" "} ({variation === "variation" ? "🅱️" : "🅰️"}) {/* Product Details */} {/* Title and Price */} //highlight-start {data.title} //highlight-end ${data.price?.toLocaleString()} {data.inStock ? ( ✅ In Stock ) : ( ❌ Out of Stock )} {/* Description */} //highlight-start {data.description} //highlight-end {/* Key Features */} {data.features && data.features.length > 0 && ( Key Features {data.features.map((feature, index) => ( ✓ {feature} ))} )} {/* Technical Specifications */} {data.specs && ( Specifications {Object.entries(data.specs).map( ([key, value]) => value && ( {key.replace(/([A-Z])/g, " $1").trim()} {value} ) )} )} {/* Purchase Button */} {data.inStock ? "🛒 Add to Cart" : "Out of Stock"} ❤️ Add to Wishlist {/* Footer */} © 2025 Camera World. All rights reserved. A GrowthBook + Sanity + Next.js Demo ); } ``` ## Testing Your Setup With your Next.js app running, you can now test the A/B testing functionality. Visit the app in your browser to see which variation is being served. Even when you refresh the page, the same variation will be served to the user. Check your console to see debug info. :::tip Clear or override the GB_UUID_COOKIE to see different variations ::: ## Next Steps - **Add Event Tracking**: Replace the `console.log` with your analytics service (Google Analytics, Segment, etc.) See docs for more details. - **Add Metrics**: Add metrics to your experiment to track conversions, revenue, etc. - **Configure Webhooks**: Set up GrowthBook webhooks to revalidate your Next.js cache when experiments change ## Troubleshooting - **No variations showing**: Check that your GrowthBook client key is correct and the experiment is published. - **Console errors**: Check that all environment variables are set correctly. - **Other issues**: Restart your Next.js app and open an incognito window to test. This setup provides a foundation for content-driven A/B testing with Sanity and GrowthBook. The integration allows content creators to manage experiment variations directly in Sanity Studio while developers control the experiment logic through GrowthBook. --- ## How to feature flag and A/B test with GrowthBook, Strapi, and Next.js: A Complete Guide # How to Feature Flag and A/B Test with GrowthBook and Strapi Want to A/B test your content without changing code for every experiment? This tutorial shows you how to integrate [Strapi](https://strapi.io/) with [GrowthBook](https://growthbook.io), enabling you to feature flag and A/B test your content directly from your CMS. In this tutorial, you'll learn how to: - Set up content in Strapi for feature flagging and A/B testing - Create feature flags and experiments in GrowthBook - Configure a Next.js frontend with Strapi and GrowthBook to dynamically serve content By the end, you'll have a robust system for easily testing content variations with GrowthBook and Strapi! ## Prerequisites Before starting, ensure you have: - A [Strapi](https://strapi.io/) project - A [GrowthBook](https://growthbook.io) account - A [Next.js](https://nextjs.org/) project with basic Strapi and GrowthBook integrations set up - See Strapi's guide and example for [integrating Strapi with Next.js](https://strapi.io/integrations/nextjs-cms). - See our guide and example for [integrating GrowthBook with Next.js](/guide/nextjs-app-router). ## Create Content in Strapi This integration is flexible enough that we can use nearly **any Strapi content** with feature flags or A/B tests. The only requirement is to return an array of objects (like components). For this tutorial, we'll create a Product collection with a **Dynamic Zone** field that holds Product Card components. [Dynamic Zones](https://docs.strapi.io/dev-docs/backend-customization/models#dynamic-zones) let us create multiple variations of content within a single collection—perfect for A/B testing different versions. On our frontend, we'll show 3 shoe product cards and use GrowthBook to A/B test which shoe is displayed in the third spot. ![Strapi A/B test example](/images/guides/strapi-ab-test-example.webp) ### Product Card Component First, let's create a Product Card component for the Dynamic Zone. In the **Content-Type Builder**, go to **Components** and create a new component called `productCard`. The fields for this component are up to you, but here's an example: - name - description - image - colors - reviews - rating - price In our frontend, we use these fields to render the Product Card component. The image below shoes the `productCard` component in Strapi and how it's rendered in the frontend. ![Strapi Product Card](/images/guides/strapi-product-card.webp) ### Product Collection Next, create a new collection called **Product**. Add a Dynamic Zone field called `products` that uses the `productCard` component. We'll create a **Shoes** product collection with 2 `productCard` components that we'll test against each other: - Control: Quantile running shoe - Variation: P-Value training shoe ![Strapi Product Collection](/images/guides/strapi-shoes-collection.webp) ## Create Feature Flags and A/B Tests in GrowthBook The next step is to create a feature flag in GrowthBook. With feature flags, we can toggle content on and off, implement advanced targeting, and use the flags to run A/B tests. In this example, we'll set up a feature flag for the **Product** collection we just created in Strapi. - Go to **Features** and click **Add Feature**. - Add `shoes` as the **Feature Key**. (This key is used to reference the feature flag in your frontend.) - Select **Number** as the value type. Set the **Default Value** to `0`. - Click **Create**. ![Creating a feature flag in GrowthBook](/images/guides/strapi-feature-flag.webp) ### Create the A/B Test The feature flag we created in the previous step allows us to easily create an A/B test. - Go to the **shoes** feature flag. - Scroll down to **Rules** and click **Add Rule**. - Select **Experiment** and click **Next**. - Add an experiment name, which can be whatever you want. Other fields are optional. - Click **Next**. ![Creating an experiment in GrowthBook](/images/guides/strapi-growthbook-experiment.webp) In this example, we'll test 2 variations, but you can add as many as you like. Click **Save → Review & Publish** when you're done. The experiment is now live and ready to be integrated in your frontend. :::info Zero-based index The value of the feature flag directly maps to the object's zero-based index in your Dynamic Zone. For example, if the feature flag value is `0`, then the first `productCard` is served. If the value is `1`, then the second variation is served and so on. ::: ## Integrate Strapi and GrowthBook in Next.js So far, we've set up Strapi and GrowthBook separately. Now, it's time to bring them together in our Next.js frontend. Make sure you've already configured your Next.js app to fetch content from Strapi and integrated with GrowthBook (see the [guides for each integration above](#prerequisites)). This section walks through how to dynamically serve content based on the A/B experiment created in GrowthBook. ### Create a Helper Function This helper function takes in your Strapi content and returns the correct variation based on GrowthBook's feature flag value, which, in our case, is the result of the A/B test. If the feature flag is toggled off, the function returns `null`. The function takes in 3 parameters: 1. `gb`: Your GrowthBook instance (created as part of your GrowthBook integration into Next.js). 2. `variations`: The content variations fetched from Strapi. 3. `featureFlagKey`: The feature flag key from GrowthBook that [we created above](#create-feature-flags-and-ab-tests-in-growthbook). In your Next.js app, add the following code: ```tsx import { type GrowthBook } from "@growthbook/growthbook"; type Props = { gb: GrowthBook; variations: T[] | null; featureFlagKey: string; }; export default function getVariation({ gb, variations, featureFlagKey, }: Props): T | null { // Evaluate the featue flag const result = gb.getFeatureValue(featureFlagKey, null); // If the feature flag is off or variations are empty, return null if (!variations || result === null) { return null; } // Return the variation based on the feature flag value return variations?.[result] ?? null; } ``` ### Update page.tsx The last part of the process is to update a component to use the variation returned by the `getVariationHelper`. ```tsx export default async function Home() { // Get the variation for the "shoes" feature flag const variation = getVariation({ gb, variations: shoesToTest, featureFlagKey: "shoes", }); return ( {/* A default collection of items to always show */} {defaultShoes.map((product: Card) => ( ))} {/* A single product, dynamically shown based on the A/B test */} ); } ``` Now, the app renders a different product card based on the A/B test result. The control shows the Quantile shoe, and the variation shows the P-Value. ![Control vs Variation product cards in Strapi, GrowthBook, and Next.js](/images/guides/strapi-control-vs-variation.webp) :::tip Tracking metrics To measure the success of your experiments, make sure to track relevant metrics (like clicks or conversions) using your analytics platform. See our guide on [setting up analytics with GrowthBook](/app/datasources). ::: ## Summary In this tutorial, you built a complete A/B testing system that combines the power of Strapi's content management with GrowthBook's experimentation capabilities 🎉 Here's a recap of what you learned: - Structure content in Strapi for A/B testing using Dynamic Zones - Create and configure feature flags in GrowthBook - Build a helper function to dynamically serve content variations - Integrate everything in a Next.js frontend Now you can launch and iterate on content experiments directly from your CMS, without needing developer help for each test! ## Next Steps Ready to take your experimentation system further? Here are some ways to build upon what you learned: - [Integrate with analytics platforms](/app/datasources) like Google Analytics or Segment. - [Set up advanced targeting](/features/targeting) for your experiments and feature flags. ## Resources - [Strapi Documentation](https://docs.strapi.io/) - [GrowthBook Documentation](/) - [Next.js Documentation](https://nextjs.org/docs) - [GrowthBook Next.js Examples](https://github.com/growthbook/examples/tree/main/next-js) - [LaunchPad: Strapi Next.js Integration Example](https://github.com/strapi/LaunchPad) ## Troubleshooting 1. **Content not updating in frontend** - Check if content is published in Strapi - Verify the Dynamic Zone array indices match your feature flag values - Clear your browser cache and reload 2. **Feature flag not working** - Ensure the feature flag key matches exactly (e.g., "shoes") - Check if the experiment is running in GrowthBook - Verify your GrowthBook SDK connection 3. **Helper function returning null** - Confirm variations array is not empty - Check if feature flag is enabled in GrowthBook - Log the feature flag value to verify it's working --- ## Importing External Experiments import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Importing External Experiments GrowthBook can analyze experiments that are run on another service or platform so long as they exist in your linked Data Source. To do this, we create a dedicated experiment in GrowthBook that links to your data warehouse via the experiment metadata. If this is your first time importing any experiments, you need to [connect your data warehouse](/warehouses) and set up your [experiment assignment queries](/app/datasources#experiment-assignment-queries) which will tell GrowthBook where to find the experiment data. Once your datasource is set up, you can import an experiment by clicking select Create Experiment from the Experiment page and then selecting Import. ## Importing Experiments When you choose to Import an Existing Experiment, we scan all of the experiment assignment queries for a datasource and present any experiments we have found in those experiment assignment queries. ![Experiment Import Modal](/images/import-experiment-modal.png) We scan the experiment assignment queries for any experiment metadata that we can use to create an experiment in GrowthBook. This metadata includes the experiment name, the variations and their splits, and the start date of the experiment. You can choose to Import an experiment, and you will be prompted to confirm this metadata. Because we have to infer the variation splits and the start date from the data we query from the experiment assignment queries, please review the metadata carefully before finalizing the import. ## Missing Experiments If you don't see your experiment in the import list, there are a few potential explanations: 1. The experiment was already imported! You can check the experiments page to see if it was imported, or deselect "Hide Imported" on the importer to see if it shows up there. 2. The experiment does not fit some of the minimum criteria set in the filter bar. You can see that it is possible to filter by minimum number of users and duration. 3. There was a delay in the data being available in your data warehouse. When you refresh the Importer, we only look at data since the the last data found by the importer. If there are many experiment assignment queries and one table is more delayed than another, it may cause some experiments not to show up. You can easily remedy this by clicking **Full Refresh** in the bottom right to re-scan the full history of data. 4. The exposure events for your experiment are not in one of the experiment assignment queries for your datasource. Ensure you're selecting the right datasource and that it has the correct query to find your experiment data. --- ## GrowthBook Documentation import ButtonCard from '@site/src/components/ButtonCard' # Introduction GrowthBook is the most popular open-source platform for feature flagging and experimentation. It's designed to help teams deliver better software faster by making it easy to test and roll out new features. :::tip Got questions? We would love to help you get started with GrowthBook. You can [meet with us](https://calendly.com/growthbook) or [join our Slack](https://slack.growthbook.io?ref=docs-home) ::: ### Quick Links } /> } /> } /> ## Our Goals The top 1% of companies spend thousands of hours building their own feature flagging and experimentation platforms in-house. We want to give all companies the flexibility and power of these in-house platforms without needing to build it themselves. - We believe that **feature flagging** is the best way to release code, and **experimentation** is the best way to measure its impact on your users. - We believe experimentation should sit on top of your **existing data and metrics**, wherever they live and however they are defined. - We believe in **data transparency**. See the SQL behind every query, export results as a Jupyter notebook, and view our [stats engines on GitHub](https://github.com/growthbook/growthbook/tree/main/packages/stats). - We are fanatical about **performance**. Our [SDKs](/lib) are crazy fast, lightweight, and evaluate everything locally with no network requests. - We believe in **open source**. GrowthBook is open source and free to use. The exact same code that powers our Cloud platform is available for you to run entirely on your own infrastructure. - We believe in **privacy & security**. We don't collect any data about your users, and you can run GrowthBook on your own infrastructure entirely behind a firewall. ## Documentation Use the menu or the **Previous**/**Next** links to navigate these docs. [Join us on Slack](https://slack.growthbook.io?ref=docs-home) if you need help, want to chat, or are thinking of a new feature. We're here to help — and to make GrowthBook even better. --- ## Insights import MaxWidthImage from '@site/src/components/MaxWidthImage'; # Insights Insights provide a variety of tools to help you understand your experimentation program at a glance and perform meta-analyses. ## Dashboard The Dashboard provides a high-level overview of the velocity, shipping rate, and impact of your experimentation program. ### Scaled Impact Scaled Impact provides an overview of how multiple experiments have influenced a key metric. All completed experiments that have the selected metric are included in this section. ![Experiment Impact](/images/insights/impact.png) **Implementation details** For all stopped experiments that are marked as "won" or "lost," we sum the scaled impact of those experiments to present a measure of total impact. Because some experiments have multiple variations, we only pick one variation per experiment: - "Won" experiments use the variant selected as the winner in the experiment metadata - "Lost" experiments use the variation that performed worst relative to the baseline. Summing the scaled impact of "Lost" experiments helps demonstrate how experimentation provides value by preventing shipping ideas that did not pan out. We also provide an opportunity to "de-bias" these estimates so long as there are at least 6 experiments that were stopped in this window, since simple sums of experiment impacts are subject to a variety of biases when considering total impact. :::caution Summed experiment impact relies on several assumptions Summed experiment impact relies on several assumptions (independence of experiments, additivity of experiment impacts, and no selection bias). However, this tool should not replace accurate impact estimation via [holdouts](kb/experiments/holdouts). ::: Details of scaled impact and de-biasing procedure **Scaled Impact** [Scaled impact](app/experiment-results#difference-types) rescales your experiment effect to answer the question: "What would the total effect be if all participants had received a specific test variant?" For example, imagine your absolute lift in an experiment is \$0.30 per user with 100 users in that variation. If your experiment got 20\% of total traffic, and it was split evenly so that each variation got 50\% of the experiment traffic, then the total potential traffic was 100 / (50\% X 50\%) = 400 users. So, the scaled impact would be \$0.30 X 400 = \$120. It is a simple rescaling that makes some assumptions, but it does allow you to compare the effects of experiments on different parts of your product (e.g. a big change to a small feature vs. a small change to a big feature might have the same scaled impact). **Total Impact**, represented by the large **Won** and **Avoided loss** boxes, sums the effects of all experiments within a category. This assumes the effects are independent and not additive. **This is not an assumption easily satisfied**, but it allows you to get a general sense of scale. **De-biasing** is achieved through the [_positive-part_ James-Stein](https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator#Improvements) shrinkage estimator, which mitigates the natural bias in experiment outcomes (where more decisive results tend to show larger effects). This estimator adjusts the results by estimating variance only from the shown experiments and shrinks all impacts toward zero. While this reduces selection bias, it does not address concerns about the independence or additivity of experiments. ### Win Percentage & Experiment Status Win percentage shows the percentage of experiments that were stopped and marked as "won." Experiment status shows the number of experiments completed in the selected time period and represents your experimentation program's velocity. ![Win Rate and Velocity](/images/insights/win-rate.png) The win percentage graph is highlighted around 33\% because this represents an ideal win rate for teams experimenting often. Higher win rates can suggest that you might want to be testing more ideas! ### North Star Metrics North Star Metrics are crucial, company-wide indicators of success. This panel displays how these metrics evolve over time and shows which experiments using this metric are running (or when they ended). If this panel isn't visible, go to **Settings** → **General** to set up your North Star metrics. ## Learnings ![Experiment Learnings](/images/insights/learnings.png) The Learnings page allows you to review all stopped experiments that match the selected filters and see: 1. The experiment decision (won, lost, inconclusive) 2. Screenshots of the winning variation 3. Additional experiment metadata The Learnings page provides a handy, in-depth overview of all recently completed experiments. The better the metadata input when creating and stopping an experiment, the richer the data on this page. ## Timeline ![Experiment Timeline](/images/insights/timeline.png) The Timeline page shows all experiments that match the selected filters, including their start and stop dates (if not still running). If an experiment had multiple phases, we place those into separate blocks on the same row. Stopped experiments are colored to indicate whether they won (green), lost (red), or were inconclusive (gold). ## Metric Effects ![Metric Effects](/images/insights/effects.png) The Metric Effects page allows you to select a metric and view a histogram of experiment impacts for that metric from running and completed experiments. This shows you the typical spread of effects across your experiments, as well as providing you with historical lift data that may be helpful if using informative priors in our Bayesian engine. ## Metric Correlations The Metric Correlations page allows you to visualize how experiments tend to jointly impact two metrics. Each dot on the plot is a variation from an experiment, with the size of the dot corresponding to the number of units and the lines corresponding to the error bars. This graph helps you answer questions like: - When my experiments increase one metric, are other metrics following suit? - Is there any trade-off between maximizing one key metric and another key metric? ![Metric Correlations - Positive Correlation](/images/insights/correlations.png) The above screenshot shows two positively correlated metrics ("Any Purchase" and "Average Order Value"). Because the dots are mostly in the top-right and bottom-left quadrants, we can see that when experiments increase the "Any Purchase" metric, they tend to also increase the "Average Order Value" metric. This means that your experiments tend to benefit or harm whether users make _any_ purchases as well as the average value of those purchases. The next screenshot shows a more ambiguous case (or, potentially, even a negative correlation): ![Metric Correlations - Negative Correlation](/images/insights/correlations-two.png) In this example, consider an app with in-app purchases. "User Retention" and "Total User Revenue" are both important as User Retention could be a leading indicator for long-run success and revenue. In this image, the effects are more scattered—or even concentrated in the top-left and bottom-right quadrants—indicating a neutral or negative correlation. This could mean there is a trade-off between maximizing revenue and keeping users engaged. This could indicate a [dark pattern](https://blog.growthbook.io/dark-patterns-a-b-testing/), where your experiments are driving purchasing behavior but somehow also making the product worse. In this situation, you'll often want to consider which goal metric is truly important and how to make trade-offs between them, or consider new experiments or features that may improve both jointly. For a more technical discussion of this topic, read this excellent blog post by Tom Cunningham: [Thinking About Tradeoffs? Draw an Ellipse](https://tecunningham.github.io/posts/2023-10-23-pareto-frontiers-experiments-ranking.html). --- ## Experiment skills These [Agent Skills](/integrations/ai-agents/agent-skills) take an A/B test from idea to decision, all from your AI agent. Install the plugin and run [`/growthbook:gb-setup`](/integrations/ai-agents/agent-skills#installation) first. The example prompts can be typed verbatim, or your agent may run these skills automatically when it detects matching intent. GrowthBook's Head of Experimentation, Luke Sonnet, runs an experiment end to end using the skills. ## Prerequisites - **A datasource and at least one metric**, [configured in the GrowthBook UI](/app/experiment-configuration). The skills reference metrics by ID. They don't create datasources or metrics. - The experiment skills target standard A/B tests. **Multi-armed bandits aren't supported yet**: the skills stop and point you to the app rather than misinterpret a bandit. ## The experiment lifecycle The five skills below form a chain from ideation to decision. Each one routes to the next when you're ready. | Skill | What it does | Example prompt | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | `experiment-brainstorm` | Propose new test ideas grounded in your team's past stopped-experiment history. Read-only. | "What should we test next on the checkout page?" | | `experiment-design` | Walk through hypothesis, variations, primary metric, guardrails, and sample size to produce a launchable spec. Read-only. | "Help me design an A/B test for the new pricing page" | | `experiment-launch` | Create the experiment, create or reuse the feature flag, wire the experiment-ref rule, and start it. | "Launch the pricing experiment we just designed" | | `experiment-analyze` | Refresh the snapshot if it's stale, then interpret results: lifts, confidence intervals, guardrails, and a data-quality (SRM) check. Read-only. | "What are the results of `exp_abc123`?" | | `experiment-stop` | Stop a running experiment, optionally declaring a winner and rolling it out to 100% of eligible traffic. | "Stop the pricing test and ship the winner" | Related concepts: [Experiment configuration](/app/experiment-configuration), [Experiment results](/app/experiment-results), [Decision framework](/app/experiment-decisions). ## From experiment to cleanup After you stop an experiment, the feature flag it ran through usually still needs tidying: inlining the winning value at your code sites and removing the flag. Hand off to [`flag-cleanup`](/integrations/ai-agents/agent-skills/feature-flags#operations), which walks through code-site inlining and a two-step archive-then-delete. ## Next steps - [Feature flag skills](/integrations/ai-agents/agent-skills/feature-flags): create, target, and clean up the flags your experiments run on - [Agent Skills overview](/integrations/ai-agents/agent-skills): install and setup - [Experimentation documentation](/app/experiment-configuration): the underlying concepts [Join our community Slack for additional tips and tricks](https://join.slack.com/t/growthbookusers/shared_invite/zt-2xw8fu279-Y~hwnfCEf7WrEI9qScHURQ) --- ## Feature flag skills These [Agent Skills](/integrations/ai-agents/agent-skills) cover the full feature flag lifecycle from your AI agent. Install the plugin and run [`/growthbook:gb-setup`](/integrations/ai-agents/agent-skills#installation) first. Every flag change in GrowthBook goes through a **draft revision** before it goes live. The skills below stage their changes on a draft and hand off to `flag-publish` to ship it: the same draft → review → publish flow you'd use in the app. See [Publishing & approval flows](/features/publishing-and-approval-flows) for the concepts. The example prompts can be typed verbatim, or your agent may run these skills automatically when it detects matching intent. ## Revision lifecycle These three skills manage the draft → review → publish flow that every other flag change flows through. | Skill | What it does | Example prompt | | ---------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------- | | `flag-revisions` | List and inspect open drafts, see who owns them and their approval status. | "What drafts are open on the `checkout-redesign` flag?" | | `flag-review` | Request a review on a draft, or approve / request changes / comment as a reviewer. | "Request review on my draft for `new-pricing`" | | `flag-publish` | Publish a draft live, resolve merge conflicts, discard, or revert to a prior revision. | "Publish the approved draft for `new-pricing`" | ## Operations Create flags and manage their metadata, default value, environment state, and lifecycle. | Skill | What it does | Example prompt | | -------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | `flag-create` | Create a new flag: collision check, value type, environments, default value. Ships disabled everywhere. | "Create a boolean flag `new-checkout-flow` defaulting to false" | | `flag-metadata` | Update a flag's description, owner, project, tags, custom fields, or JSON schema. | "Tag `dark-mode` as `ui` and set me as the owner" | | `flag-default-value` | Change the fallback value served when no targeting rules match. | "Change the default value of `max-upload-mb` to 50" | | `flag-toggle` | Enable or disable a flag in a specific environment. | "Turn off `new-checkout-flow` in production" | | `flag-prerequisites` | Gate an entire flag on another boolean flag being on. | "Only evaluate `beta-dashboard` when `beta-program` is on" | | `flag-cleanup` | Archive or delete a stale flag, walking through code-site inlining first. | "Clean up the `old-banner` flag" | Related concepts: [Feature flag basics](/features/basics), [Stale flag detection](/features/stale-detection), [Code references](/features/code-references). ## Rules Control who gets which value, and when. For a new A/B test, see also the [Experiment skills](/integrations/ai-agents/agent-skills/experiments). | Skill | What it does | Example prompt | | ----------------- | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | `flag-rules` | List rules in evaluation order, delete a rule, reorder, or route to the right rule skill. | "Show me the rules on `new-checkout-flow`" | | `flag-targeting` | Add, edit, or remove force and percentage-rollout rules, with conditions and saved groups. | "Roll out `new-checkout-flow` to 10% of US users" | | `flag-schedule` | Time-gate a rule with a start and/or end datetime for automatic activation. | "Enable the sale banner Friday at 9am and turn it off Monday" | | `flag-ramp` | Multi-step ramp schedule that increases coverage over time, with per-step intervals or gates. | "Ramp `new-checkout-flow` from 10% to 100% over a week" | | `flag-monitoring` | Monitored "safe rollout": a ramp with guardrail metrics and optional auto-rollback. | "Roll out `new-checkout-flow` safely and watch checkout latency" | | `flag-experiment` | Add an experiment-ref rule to run an A/B test through the flag. | "Run an experiment through `new-checkout-flow`" | Related concepts: [Targeting & rules](/features/rules), [Targeting conditions](/features/targeting), [Safe rollouts](/features/safe-rollouts), [Environments](/features/environments). ## Discovery Find and audit flags, and trace how they depend on one another. Both skills are read-only. | Skill | What it does | Example prompt | | ------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------- | | `flag-search` | Search, list, and audit flags by project, tag, owner, environment state, or staleness. | "List all stale flags owned by me" | | `flag-graph` | Trace a flag's prerequisites, its dependents, and any linked experiments and holdouts. | "What depends on `beta-program`? Is it safe to delete?" | ## Next steps - [Experiment skills](/integrations/ai-agents/agent-skills/experiments): design, launch, analyze, and stop tests - [Agent Skills overview](/integrations/ai-agents/agent-skills): install and setup - [Feature flag documentation](/features/basics): the underlying concepts [Join our community Slack for additional tips, help, and announcements](https://join.slack.com/t/growthbookusers/shared_invite/zt-2xw8fu279-Y~hwnfCEf7WrEI9qScHURQ) --- ## GrowthBook Agent Skills import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; [Agent Skills](https://docs.claude.com/en/docs/agents-and-tools/agent-skills/overview) are folders of instructions, scripts, and resources that an AI agent loads on demand to carry out a specialized task. Anthropic introduced the format, and it works in any agent that follows the [Agent Skills standard](https://agentskills.io), including Claude Code, Cursor, and Codex. GrowthBook publishes a library of them that runs the entire feature flag and experiment lifecycle from your AI coding agent (create a flag, roll it out, run an experiment, read the results, and clean up afterward) without leaving your editor and **without running an MCP server**. Each skill is a playbook your agent follows: it knows the GrowthBook REST API, the right order of operations, and the guardrails (draft → review → publish, safe rollouts, two-step deletes) that keep changes safe. :::info Open source The skills live at [github.com/growthbook/skills](https://github.com/growthbook/skills) and call the [GrowthBook REST API](/app/api) directly through a small bundled helper. ::: Watch GrowthBook's Head of Experimentation, Luke Sonnet, run an experiment end-to-end using skills. ## Prerequisites - **A GrowthBook account:** [GrowthBook Cloud](https://app.growthbook.io) or a self-hosted instance. - **A Personal Access Token (PAT):** create one at [**Account** → **Personal Access Tokens**](https://app.growthbook.io/account/personal-access-tokens). The token is tied to your GrowthBook user, so flags and experiments the skills create are attributed to you automatically. - **Node.js 18+:** check with `node -v`. Many agents already run on it. ## Installation ### 1. Install the plugin Add the marketplace, then install the plugin: ```text /plugin marketplace add growthbook/skills /plugin install growthbook@growthbook-skills ``` Install with the [Agent Skills](https://agentskills.io) CLI: ```bash npx skills add growthbook/skills ``` Restart your agent if the skills don't appear right away. ### 2. Configure your credentials Run the setup skill and follow the prompts: ```text /growthbook:gb-setup ``` It asks for your PAT or API key (and, for self-hosted, your API URL), validates them against the live API, and writes `~/.config/growthbook/.env` with `chmod 600`. Every other skill reads that file automatically. :::tip Prefer environment variables? The skills read environment variables first and only fall back to the file. Export them instead if you like. Handy for CI: ```bash export GB_API_KEY= # required: PAT or Secret Key export GB_API_URL=https://api.your-host # self-hosted only ``` ::: ### 3. Verify ```text /growthbook:flag-search ``` This should list your existing feature flags. If something's off with the credentials, the error points you back to `/growthbook:gb-setup`. ## How skills work Skills fire two ways: - **Automatically**, when your agent detects matching intent: "create a feature flag for the pricing page" runs `flag-create`; "what should we test next?" runs `experiment-brainstorm`. - **Explicitly**, by typing the slash command, e.g. `/growthbook:flag-search` or `/growthbook:experiment-launch`. Because each skill names its trigger phrases and routes to sibling skills, they compose cleanly when chained: - **Flag-first:** `flag-create` → `flag-toggle` → `flag-targeting` → `flag-ramp` / `flag-monitoring` → `flag-cleanup` - **Experiment-first:** `experiment-design` → `experiment-launch` → `experiment-analyze` → `experiment-stop` → `flag-cleanup` - **Experiment on an existing flag:** `flag-experiment` → `experiment-launch` → `experiment-stop` → `flag-cleanup` ## What's included The skills fall into three groups. The two reference pages list each one with an example prompt you can run. - **Setup** — `gb-setup` configures and validates your API credentials. - **[Feature flag skills](/integrations/ai-agents/agent-skills/feature-flags)** — create, target, roll out, ramp, monitor, review, publish, search, and clean up flags across their full lifecycle. - **[Experiment skills](/integrations/ai-agents/agent-skills/experiments)** — brainstorm, design, launch, analyze, and stop A/B tests. ## Example prompts - "Create a boolean flag `new-checkout-flow` that defaults to `false`, then roll it out to 10% of US users." - "What feature flags are stale and safe to clean up?" - "Design an A/B test for the new pricing page and launch it on the `pricing-v2` flag." - "What are the results of our checkout experiment? Should we ship the winner?" - "Stop the `homepage-hero` experiment and roll the winning variation out to everyone." ## Safety and governance The skills follow GrowthBook's built-in safeguards: - **Draft → review → publish.** Every flag change is staged on a draft revision. Skills route through `flag-review` and `flag-publish` so approvals and merge conflicts are handled the same way they are in the app. See [Publishing & approval flows](/features/publishing-and-approval-flows). - **Two-step deletes.** `flag-cleanup` archives a flag, pauses for you to verify nothing broke, and only then deletes. It won't collapse those steps. - **Read-only by default where it matters.** `flag-search`, `flag-graph`, `experiment-brainstorm`, `experiment-design`, and `experiment-analyze` never write. - **Secrets stay local.** `gb-setup` writes your key to a `0600` file and recommends a scoped PAT you can revoke independently. ## Next steps - [Feature flag skills](/integrations/ai-agents/agent-skills/feature-flags): the full flag catalog - [Experiment skills](/integrations/ai-agents/agent-skills/experiments): design, launch, analyze, and stop tests - [Agent Skills vs. the MCP Server](/integrations/ai-agents) - [GrowthBook REST API](/app/api), which the skills call under the hood - [Skills source on GitHub](https://github.com/growthbook/skills) [Join our community Slack for additional help, tips, and announcements](https://join.slack.com/t/growthbookusers/shared_invite/zt-2xw8fu279-Y~hwnfCEf7WrEI9qScHURQ) --- ## Connect AI Agents to GrowthBook GrowthBook works directly inside your AI coding agent: Claude Code, Cursor, and others. There are two ways to connect, and they're complementary: - **[MCP Server](/integrations/mcp):** a live set of tools your agent can call mid-conversation. - **[Agent Skills](/integrations/ai-agents/agent-skills):** installable workflows that run the full feature-flag and experiment lifecycle, with GrowthBook's draft → review → publish guardrails built in. No server to run. You can install both. Many teams reach for the MCP server for quick, conversational tasks and the skills for repeatable, end-to-end workflows. ## Which should I use? | | MCP Server | Agent Skills | | ---------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | **What it is** | A server exposing GrowthBook tools to your agent | Markdown playbooks your agent follows step by step | | **Setup** | Add an MCP server entry that runs `npx @growthbook/mcp` | Install the plugin (no running server) | | **Best for** | Quick lookups and one-off actions while you code | Repeatable, multi-step lifecycle workflows | | **Lifecycle coverage** | Core flag, experiment, and product analytics actions | The full flag and experimentation lifecycle: create, target, ramp, experiment, analyze, stop, and clean up | | **Governance** | Direct API calls | Draft → review → publish, safe rollouts, and two-step deletes built into each skill | | **Where it runs** | Any MCP-compatible host, including agents that can't run shell commands | Agents that can run shell commands: Claude Code, Cursor, Codex, and other [agentskills.io](https://agentskills.io) agents | :::info Agent can't run a terminal? Use the MCP server The Agent Skills work by having your agent run a local Node helper, so they only work in agents that can execute shell commands, such as Claude Code, Cursor, and Codex. The MCP server is launched by your agent host from a one-time config entry. That lets it work in MCP-compatible hosts that don't give the agent a terminal, like Claude Desktop. If your agent can't run shell commands, use the MCP server. ::: :::tip Not sure where to start? Start with [Agent Skills](/integrations/ai-agents/agent-skills) if you want to run flags and experiments end-to-end from your agent. Add the [MCP Server](/integrations/mcp) if you mostly want quick, conversational access while you code. ::: ## Next steps - [Set up the MCP Server](/integrations/mcp) - [Install the Agent Skills library](/integrations/ai-agents/agent-skills) - [Browse the GrowthBook REST API](/app/api), which both tools call under the hood [Join our community Slack for additional help and announcements](https://join.slack.com/t/growthbookusers/shared_invite/zt-2xw8fu279-Y~hwnfCEf7WrEI9qScHURQ) --- ## AI Integrations import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; # AI Integrations GrowthBook supports AI integrations to enhance your experimentation and feature management capabilities. These integrations allow you to leverage AI models for various tasks, such as generating insights, automating decisions, and improving user experiences. AI features in GrowthBook are optional and can be enabled or disabled in the AI settings page. You can also configure the prompts used by the AI models to better suit your needs. ![AI Settings](/images/integrations/ai/ai-settings-cloud.png) :::tip Prompts customization Many of the prompts used in AI integration in GrowthBook can be adjusted from the AI settings page. You can customize the prompts to better suit your needs and the context of your experiment program. ::: ## Available AI Features ### Generate experiment results summaries When you stop an experiment, you can generate a summary of the results using OpenAI's language models. This summary can help you quickly understand the impact of the experiment and make informed decisions. The prompt for generating this summary can be adjusted from the settings page of the OpenAI integration. #### Data sent: - Experiment variation names - Experiment metrics and their values for this experiment - Experiment metadata (e.g., start date, end date, etc.) - Experiment results (e.g., p-value, confidence level, etc.) - Experiment description and hypothesis - Experiment tags and attributes ### Generate SQL for SQL Explorer You can use AI to generate SQL queries for the SQL Explorer in GrowthBook. Add you provide a natural language description of the data you want to query, and the AI will generate the SQL query for you. ![SQL Generation](/images/integrations/ai/sql-generation.png) #### Data sent: - Table names and columns in your database - Your description of the query you want to generate - Current SQL query (if any) ### Generate Metric Descriptions You can also use AI to generate descriptions for your metrics. This can help your team better understand what a metric is used for when adding a metric to an experiment. It will take the metric SQL and meta information and generate a description. #### Data sent: - Metric name - Metric SQL - Metric attributes (e.g., type) - Metric tags ### Hypothesis checking Experiment hypothesis are crucial for guiding your experiments. With this AI integration, you can check the validity of your experiment hypothesis to ensure consistency across your experiment program. This feature analyzes the hypothesis against an adjustable prompt and provides feedback and suggestions for improvement. #### Data sent: - Experiment hypothesis text ### Similar Experiment Detection You can use AI to detect similar experiments in your GrowthBook project. This feature helps you identify experiments that may redundant, ensuring that you don't waste resources on testing similar ideas. The AI model analyzes the experiment metadata and provides a list of similar experiments. ![Similar Experiment Detection](/images/integrations/ai/similar-experiments.png) This feature is available from the "new experiment" modal when you create a new experiment. It will search the description and hypothesis and show you a list of similar experiments. This similar experiment detection is based on vector embeddings, which means it can find similar experiments even if they don't share the exact same words. The embeddings will update automatically when the experiment meta information changes, however, you can regenerate the embeddings for your experiments at any time from the AI settings page. #### Data sent: - Experiment description - Experiment hypothesis - Experiment name ## Self-Hosted AI Integrations Self-hosted GrowthBook instances can enable these AI features, by adding a OpenAI API key to the AI settings page. To get started with the OpenAI integration in GrowthBook, follow these steps: 1. **Create an OpenAI Account**: If you don't have an OpenAI account, sign up at [OpenAI](https://www.openai.com/). 2. **Obtain API Key**: After creating your account, obtain an API key from the OpenAI dashboard. This key will be used to authenticate your requests to the OpenAI API. 3. **Add environment variable**: Add the OpenAI API key to your GrowthBook instance as an environment variable. The variable should be named `OPENAI_API_KEY`. For example, if you are using Docker, you can add the following line to your `docker-compose.yml` file: ```yaml environment: - OPENAI_API_KEY=your_openai_api_key_here ``` If you are running GrowthBook in a different environment, refer to your platform's documentation on how to set environment variables. 4. **Configure GrowthBook**: In your GrowthBook settings, navigate to the AI settings tab and enable AI OpenAI integration, and configure any additional prompts as desired. From the settings page you can also select the AI model you want to use. The default is `gpt-4o-mini`. ![AI Settings Self-hosting](/images/integrations/ai/ai-settings-self-hosting.png) You can learn more from the [OpenAI Documentation](https://platform.openai.com/docs/). --- ## Microsoft Clarity (Heatmaps) # Microsoft Clarity Heatmaps This guide walks through how to integrate GrowthBook A/B tests with Microsoft Clarity for getting heatmaps of your experiment pages. This will work with GrowthBook client side SDKs (HTML or Javascript). ## Passing Data to Microsoft Clarity To make the heat maps aware of your A/B tests, you should send the experiment id and variation id to Clarity. This can be done through the "trackingCallback" in the GrowthBook client, by setting a custom tag in Clarity. You can use either one event for both experiment name and variation id, or two separate custom events. eg: ```javascript // two separate events window.clarity('set', 'experiment', experiment.key); window.clarity('set', 'variation', result.key); // one event with both, with the name stored as the 'name' and variation as the 'value' window.clarity('set', experiment.key, result.key); ``` This callback may be different depending on your implementation. Here are some full examples: ### Javascript If you are using a recent version (> 1.4.1) you can use the third party tracking plugin: ```javascript import { thirdPartyTrackingPlugin } from "@growthbook/growthbook/plugins"; // Optional settings for the plugin const pluginOptions = { // By default, it will attempt to send to all 3 of these trackers: ["segment", "ga4", "gtm"], // Additional custom tracking callback additionalCallback: (experiment, result) => { if (window?.clarity) { window.clarity('set', 'experiment', experiment.key); window.clarity('set', 'variation', result.key); } } } const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", plugins: [ thirdPartyTrackingPlugin(pluginOptions), ], }); ``` If you have an older SDK, you can manually send the data to Clarity in the trackingCallback: ```javascript const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", trackingCallback: (experiment, result) => { // existing tracking callback here. if (window?.clarity) { window.clarity('set', 'experiment', experiment.key); window.clarity('set', 'variation', result.key); } } }); ``` ### HTML SDK This example assumes you're using Google Analytics/GTM. Place this before you load the GrowthBook HTML SDK: ```html window.growthbook_config = window.growthbook_config || {}; window.growthbook_config.trackingCallback = (experiment, result) => { if (window.gtag) { window.gtag("event", "experiment_viewed", { experiment_id: experiment.key, variation_id: result.key, }); } else { window.dataLayer = window.dataLayer || []; window.dataLayer.push({ event: "experiment_viewed", experiment_id: experiment.key, variation_id: result.key, }); } if (window?.clarity) { window.clarity('set', 'experiment', experiment.key); window.clarity('set', 'variation', result.key); } }; ``` If you are using another tracker, you can replace the `gtag` and `dataLayer` calls with the appropriate tracking code. ## Using Custom Events in Clarity You can apply custom filters to the heatmap filters in Clarity to only show data for specific experiments or variations. This can be done by creating a custom event in Clarity and filtering by that event. You can read more about this [from the microsoft site](https://learn.microsoft.com/en-us/clarity/filters/clarity-filters#filtering-with-custom-filters). --- ## DataDog Integration # DataDog Integration ## DataDog RUM with Feature Flag Data Integrating DataDog's Real User Monitoring (RUM) with GrowthBook's feature flag data lets you quickly identify which experience your users were exposed to and if the changes introduced affect the user experience. This guide will cover the initial setup and an advanced example of how to leverage DataDog and GrowthBook's powerful APIs to create a dynamic system that automatically toggles GrowthBook feature flags based on error thresholds. ### Setup :::info Older DataDog RUM browser SDK versions If your DataDog RUM browser SDK version is below v5.17.0, [enable feature flags during RUM SDK initialization](https://docs.datadoghq.com/real_user_monitoring/guide/setup-feature-flag-data-collection/?tab=browser#setup). ::: We can report all feature flag evaluations to DataDog using GrowthBook’s SDK `onFeatureUsage` callback, as shown in the snippet below. ```javascript const gb = new GrowthBookClient({ onFeatureUsage: (featureKey, result) => { datadogRum.addFeatureFlagEvaluation(featureKey, result.value); } }); ``` ### Advanced example Now that you're passing Feature Flag data to DataDog, you can build monitors and alerts when the error rate between Feature Flags variations cross a specific threshold within a pre-defined time period. #### 1. Create a DataDog monitor The beta Feature Flag Tracking feature allows you to see error rates between the various feature flag states. ![Screenshot showing error rates between feature flag variations in DataDog RUM interface](../../static/images/integrations/datadog/1-view-feature-flags-error-rates-in-data-dog.png) You can then save the Error rate view to a Dashboard. ![Screenshot adding feature flags errors per view to a DataDog dashboard](../../static/images/integrations/datadog/2-add-feature-flags-to-dashboard-data-dog.png) And from the Dashboard, there is a handy “Create Monitor” option that can seed much of the monitor's search query for you. ![Screenshot showing the "Create Monitor" option in a DataDog dashboard](../../static/images/integrations/datadog/3-create-data-dog-monitor-from-dashboard.png) This is the out-of-the-box search query created from the Create Monitor link above. You could take it a step further and calculate the difference in error rates between the Feature Flag variations (e.g. error rate when the Feature Flag is enabled vs disabled) and use that value as your alert threshold. ![Screenshot customizing a monitor query in DataDog](../../static/images/integrations/datadog/4-customize-monitor-query-in-data-dog.png) #### 2. Create a DataDog webhook Once you've defined a monitor's search query, you can then build an alert that uses DataDog's Webhooks to make a request to the GrowthBook API. Before you can call a [Webhook](https://docs.datadoghq.com/integrations/webhooks/) via a monitor alert, you'll need to build the Webhook within DataDog. You can do that via the Integrations tab. ![Screenshot showing a DataDog tile integration to create a webhook](../../static/images/integrations/datadog/5-create-datadog-webhook.png) Once installed, you can build the Webhook that will turn the feature flag off for the desired environments. Remember to update the Webhook URL with your feature flag ID [as described here](/api#tag/features/operation/toggleFeature). ![Screenshot customizing a webhook in DataDog to make a request to GrowthBook feature toggle endpoint](../../static/images/integrations/datadog/6-customize-webhook-to-hit-api-url.png) #### 3. Update the monitor to trigger the webhook Now that you've built the Webhook, you can go back to the monitor alert and instead of emailing a team member, you can configure the alert to call the Webhook. --- ## Discord integration import ExternalLink from '@site/src/components/ExternalLink' # Discord integration The GrowthBook Discord integration allows you to receive alerts for the events that you care about in a Discord channel of your choosing. ## Making A Discord Webhook The first step is to create an event webhook on your discord server: 1. Open your **Server Settings** and head into the **Integrations** tab: 2. Click the **Create Webhook** button to create a new webhook! ![](../../static/images/webhooks/discord/discord-new-webhook.png) Once your discord webhook is configured, you can copy its url: ![](../../static/images/webhooks/discord/discord-copy-webhook-url.png) ## Add the Discord integration to GrowthBook Next step is to login to the GrowthBook app and visit the **Webhooks** tab under **Settings**, also available [here](https://app.growthbook.io/settings/webhooks). You will need privileges to manage webhooks in order for this menu item to be available. Click the **Create an Event Webhook** button. You should see a modal pop up with some fields for configuring the Discord notification webhook. ![](../../static/images/webhooks/discord/new-webhook.png) Then, configure the following: - **Name**: The name of the integration. In case you have multiple integrations, this can help you tell them apart. This will also show in the contextual text alongside the alerts. - **Endpoint URL**: Copy and paste the webhook URL provided by your discord app. - **Payload**: Select the `Discord` payload. - **Event filters**: You can optionally filter by events you care about. For example, if you only care about when features are deleted, you can choose `feature.deleted` from the list. If you care about all events, leave this blank. - **Environment filters**: You can optionally choose to filter by environment. For example, if you only want to hear about events that are for the production environment, you can choose `production` from the list. For all environments, leave this blank. - **Project filters**: You can optionally choose to filter by project. For example, if you have a project named "Onboarding V2" and you only want to alert for that project, you can choose that project from the projects list. For all projects, leave this blank. - **Tag filters**: You can optionally choose to filter by tag. For all events regardless of tag, leave this blank. After configuring all the fields, press **Create** to save your new Discord notification webhook. You can also edit these fields at any point if you make a mistake. ![](../../static/images/webhooks/discord/edit-webhook.png) ### Adding more alerts If you'd like to be alerted in another Discord channel, you can add another Incoming Webhook in Discord. Then, you can create new integrations in the GrowthBook dashboard, specifying all of the same information except adding your new webhook URL. ## Testing your alerts You are now ready to test your alerts. First, you can hit the `Test` button on the webhook settings page. This should trigger a test notification. Next, perform one of the actions you're watching if you've added Event filters. If you haven't added any event filters, the quickest way to test it's working is to either create a new test feature (then delete it if it's not needed), or toggle an environment on or off for an existing feature. --- ## Official GrowthBook for Framer Plugin import MaxWidthImage from '@site/src/components/MaxWidthImage'; ## Overview The GrowthBook Framer plugin makes it easy to run A/B tests right inside your Framer projects. Assign visitors to different versions of your design (like layouts, headlines, or calls to action), track results, and confidently choose the best experience for your audience. ### How It Works Here’s a high-level overview: 1. **Create an experiment in GrowthBook**: Define your variants and set your goals—like increasing clicks or signups. 2. **Connect the experiment to Framer**: Use the GrowthBook plugin to assign different components to each variant. 3. **Track and measure**: Track which variant users see using GA4, then analyze the results in GrowthBook (via BigQuery). ### Requirements You'll need: - A [**GrowthBook account**](https://growthbook.io/) (free and open source!) - **Google Analytics 4** (GA4) set up on your Framer site. Framer has a [step-by-step guide](https://www.framer.com/help/articles/how-to-set-up-google-analytics/) for setting up GA4. - A **BigQuery** project linked to your GA4 account (for analyzing results) :::tip Need help setting up GA4 or BigQuery? Check out our [Google Analytics 4 and BigQuery docs](/guide/GA4-google-analytics). ::: ## Installation - Install the [GrowthBook plugin from the Framer Marketplace](https://www.framer.com/marketplace/plugins/growthbook/). - In GrowthBook, go to **SDK Configuration** → **SDK Connections** and click **Add SDK Connection**. Call the connection something like "Framer" and choose **React** as the SDK Language (don’t worry—this works even if you’re not using React directly). - Copy the **Client Key** from GrowthBook, paste it into the Framer plugin **Settings**, and click **Save**. ![SDK connection page in GrowthBook](/images/integrations/framer/framer-sdk-connection.png) ![Framer plugin settings](/images/integrations/framer/framer-plugin-settings.png) The plugin is now ready to use! ## Launch an Experiment In GrowthBook, feature flags are the underlying mechanism for experiments. You can think of flags like trucks and A/B tests as the cargo they carry. (Other types of cargo might include targeting rules, progressive rollouts, and more.) Now that you have the plugin installed, we'll walk through the steps to launch an experiment. ### Create a Feature-Flag Experiment in GrowthBook - In GrowthBook, go to **Features** and click **Add Feature**. - Give the feature a name and for the **Value Type**, choose `Number`, change the **Default Value when Enabled** to `0`, and click **Create**. - Under Rules, click **Add Rule** and choose **Experiment**. - Follow the prompts to create a new experiment, adding more variants as needed. - Under **Data Source**, select your data source and choose **Goal Metrics** that you'd like to test against. (You can choose or change this later.) - Click **Review & Publish** to make the experiment live. ![Feature creation page](/images/integrations/framer/framer-create-feature.png) ![GrowthBook experiment setup](/images/integrations/framer/framer-gb-experiment.png) ### Configure the Experiment in Framer - In Framer, open the GrowthBook plugin and select the feature flag you created in the previous step. (Click the refresh icon if you don't see it.) - Click **Add to Canvas** to add the GrowthBook component. - Connect the **Control** to your default component. - Connect additional components for the **Variants**. - Add the GrowthBook component to your site. - Publish your changes and you're live! ![Framer add to canvas](/images/integrations/framer/framer-plugin-add-to-canvas.png) ![Framer experiment](/images/integrations/framer/framer-experiment-setup.png) ## FAQ ### How can I see if the experiment is working? Use [GrowthBook Dev Tools](/tools/chrome-extension) to test, debug, and override experiments. The Chrome/Firefox extension makes it easy to understand and test your A/B experiments. ### Why do I need to use a numeric feature flag? The Framer plugin uses numbers to choose between your components, where `0` is your control (default). `1` is your first variant, `2` is your second, and so on. These numbers match the order of the variants in the GrowthBook experiment. ### Which targeting attributes are supported? [Target specific users or devices](/features/targeting) for your experiment by using targeting attributes, like showing a certain element to users on mobile. Several targeting attributes are supported out of the box: - `id` - `url` - `path` - `host` - `query` - `pageTitle` - `deviceType` (mobile or desktop) - `browser` (chrome, edge, firefox, safari, or unknown) - `utmSource` - `utmMedium` - `utmCampaign` - `utmTerm` - `utmContent` See the docs for how to [create custom targeting attributes](/lib/script-tag#adding-custom-attributes). ### How do I track experiment results? The plugin [automatically sends exposure events](/lib/script-tag#tracking-experiment-views) for GA 4, Google Tag Manager (GTM), and Segment. GTM users will need to add a new trigger. We have a full walkthrough on how to configure this in our [GTM docs](/guide/google-tag-manager-and-growthbook). :::note Other event trackers and data warehouses We support other event trackers and data warehouses, but they'll require some additional setup. Get in touch with us to learn more. ::: ### How do I export the experiment results to BigQuery? You'll need to connect your GA 4 property (or data warehouse) to BigQuery. See the [BigQuery docs](/guide/bigquery) for more info. ### What if I have more questions? Join our [Slack community](https://join.slack.com/t/growthbookusers/shared_invite/zt-2xw8fu279-Y~hwnfCEf7WrEI9qScHURQ) to ask questions and get help from the GrowthBook team and other users. --- ## GitHub (Metric Definitions) # Use GitHub to Version Control Metric Definitions This guide walks through how to make GitHub the source-of-truth of your metric definitions in GrowthBook. ## Setting Up GrowthBook For this guide, you can either use GrowthBook Cloud at https://app.growthbook.io or your own self-hosted deployment. ### Connect to a Data Source The easiest way to start is to use the built-in sample data in GrowthBook. Go to **Experiments** and in the Setup Steps, select the **View Sample Experiment** option. This will connect to a sample Postgres database and create a few sample metrics and an experiment. If you have your own data warehouse, you can use that instead, but you will have to change the SQL queries further in this guide to match your specific data structure. After connecting, you will need to find the data source id. The easiest way to find this is to go to **Metrics and Data** → **Data Sources**, click into the data source and copy the id from the last part of the URL. It will start with `ds_`, for example `ds_abc123`. Keep note of this id since you will need it later. ### Create an API Key Go to **Settings** → **API Keys** and click to create a new Secret Key. Make sure to select the "Admin" role since we will need to use it to create metrics. Keep note of this key (it will start with `secret_admin_`), you will need it later. ## Setting up GitHub First, create an empty GitHub repository. This is where we will define our metrics. ### Create a `metrics.yml` File To keep things simple, we will store all of our metric definitions in a single YAML file. We're going to use Fact Tables to define our metrics. This lets us write SQL once and re-use that across multiple metric definitions. We will create a single Fact Table for `orders` and 2 metrics - `purchased` and `revenue`. :::info File format requirements The format we're using below exactly matches what the GrowthBook API expects. If you want to store metrics in a different format, you will need to transform it first before sending to GrowthBook. ::: Create a new file in your GitHub repository called `metrics.yml` with the following contents. Replace `ds_abc123` with the id of your actual data source that you noted earlier. ```yml factTables: - id: orders data: name: Orders datasource: ds_abc123 userIdTypes: - user_id sql: > SELECT userId as user_id, amount, received_at as timestamp FROM orders factMetrics: - id: purchased data: name: Purchased metricType: proportion numerator: factTableId: orders - id: revenue data: name: Revenue metricType: mean numerator: factTableId: orders column: amount ``` ### Creating a Helper Script Create a file named `growthbook_sync.mjs` in your repo with the following contents (make sure to use the `mjs` extension). If you are self-hosting GrowthBook, replace `https://api.growthbook.io` with your self-hosted API host. ```js import { parse } from "yaml"; import fs from "fs"; // Edit these constants as needed const FILE_NAME = 'metrics.yml'; const API_HOST = 'https://api.growthbook.io'; const GB_API_KEY = process.env.GB_API_KEY; // Parse the yaml file const file = fs.readFileSync(FILE_NAME, 'utf8'); const json = parse(file); // Send to GrowthBook const res = await fetch(`${API_HOST}/api/v1/bulk-import/facts`, { method: "POST", headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${GB_API_KEY}` }, body: JSON.stringify(json) }); // Handle errors and print the response const resJson = await res.json(); if (!res.ok) throw new Error(resJson?.message || "Error syncing"); console.log("Success!", resJson); ``` This script does 3 things: 1. Parse the `metrics.yml` file into JSON 2. Send this payload to the GrowthBook API's `/bulk-import/facts` endpoint 3. Handle errors and print the response for easier debugging if things go wrong ### Create a GitHub Secret The script above follows best practices by using environment variables to store secrets. For this to work, you need to add a secret to GitHub. In your GitHub repository settings, create a new secret called `GB_API_KEY` and for the value, use the API Key you noted earlier (starting with `secret_admin_`). ### Set up GitHub Actions We want the script above to run every time the `metrics.yml` file changes. We can do this using GitHub Actions. Create a file in your repository named `.github/workflows/growthbook_sync.yml` with the following contents: ```yml name: Sync to GrowthBook on: push: branches: - main jobs: growthbook_sync: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20.x - name: Sync to GrowthBook run: | npm install yaml GB_API_KEY="${{secrets.GB_API_KEY}}" node growthbook_sync.mjs ``` ## Testing it Out After committing the above files to your GitHub repo, it should run automatically and you should see your new Orders fact table and 2 new metrics within your GrowthBook account. If it didn't work, look at the output of your GitHub Action to see if there are any error messages. ### Making Changes Fact Tables and Metrics created via this `bulk-import` endpoint are marked as "Official" by default. This means they cannot be edited from within the GrowthBook UI and must be changed within GitHub instead. This helps avoid things getting out-of-sync. :::note Metric creation You can still use the GrowthBook UI to create or edit other fact tables and metrics. Only "Official" ones created through this `bulk-import` endpoint will be locked down. ::: Let's try updating the `metrics.yml` file, maybe by adding a description to the Purchased metric: ```yml # ... data: name: Purchased description: Percent of people who purchased something metricType: proportion # ... ``` Commit this change to your `main` branch and GrowthBook's copy of this metric will be updated to match, usually within 30 seconds! ## Next Steps Read the API Docs on the [bulk-import/facts endpoint](/api#tag/fact-tables/operation/postBulkImportFacts) to see all of the options and fields you can use in your `metrics.yml` file. --- ## GrowthBook integrations # GrowthBook integrations GrowthBook plans on supporting more integrations in the near future. If you have a suggestion for an integration, please let us know in our [Community Slack](https://slack.growthbook.io?ref=docs-integrations) or via email at support@growthbook.io. ## Integrations list - [Microsoft Clarity](/integrations/clarity) - [Datadog](/integrations/datadog) - [Discord](/integrations/discord) - [GitHub Metrics](/integrations/github-metrics) - [Jira](/integrations/jira) - [SCIM](/integrations/scim) - [Shopify](/integrations/shopify) - [Slack](/integrations/slack) - [Webflow](/integrations/webflow) - [WordPress](/integrations/wordpress) --- ## Official GrowthBook for Jira Integration # Jira Integration ## Overview The **GrowthBook for Jira** integration connects your feature flags and experiments directly to Jira issues, streamlining your workflow, reducing context switching, and helping your team deploy confidently. What you’ll learn in this guide: - What the GrowthBook Jira integration does - How to install and configure the integration - How to link feature flags and experiments to Jira issues - How to view and manage linked feature flags and experiments within Jira - How to enable the Custom Field for your Jira Projects ## What does the GrowthBook Jira integration do? GrowthBook for Jira makes it easy for your team to integrate GrowthBook feature flags and experiments directly within your Jira workflow. Your team can quickly associate feature flags and experiments to Jira issues, view critical details instantly, and streamline communication between development, product, and QA teams. :::warning Only supported for cloud-hosted GrowthBook accounts The app listed in the Atlassian Marketplace only supports cloud-hosted GrowthBook accounts. For self-hosted instances, see the instructions in our [readme](https://github.com/growthbook/jira-app/tree/main?tab=readme-ov-file#connecting-to-a-self-hosted-growthbook-instance) ::: ## Install the Integration 1. Install the GrowthBook Jira Integration from the [Atlassian Marketplace](https://marketplace.atlassian.com/apps/1237621/growthbook-for-jira?tab=overview&hosting=cloud). 2. Click **Get it now** and follow Jira’s prompts to complete installation. 3. After installing, navigate to your Jira project’s Apps settings and select the GrowthBook Integration. 4. Connect your GrowthBook workspace by providing your GrowthBook API key and other required configuration details. :::tip Ensure you have administrator privileges in both Jira and GrowthBook to complete the setup. ::: ## (Optional) Add the GrowthBook Custom Field to Jira Projects 1. In the sidebar, click **Projects** → **Manage Projects** 2. For each project that you want to enable the custom field in, select the "..." more options menu 3. Select **Fields** and **Add field**, then search for "GrowthBook Link" and click **Add field** 4. Under **Work types**, select which work type you want to add the field to, e.g. **Task** 5. Move the GrowthBook Link field from the list of fields in the sidebar to where you'd like it to appear on your work item Once the field has been added, it will appear when you link an issue to a GrowthBook feature or experiment. ![Adding field in project settings](/images/integrations/jira/add-field-to-project.png) ![Adding field to work item](/images/integrations/jira/add-field-to-task.png) ## Adding Feature Flags and Experiments to Jira Issues ![Select menu for GrowthBook for Jira](/images/integrations/jira/jira-select-features.jpg) :::info Only existing items will be shown If the dropdown is empty, make sure you've already created a feature flag or experiment in GrowthBook. Only existing items will be shown. ::: To link a GrowthBook feature flag or experiment to a Jira issue: 1. Open the Jira issue 2. Select the gear icon (View app actions) at the top and from the dropdown choose **GrowthBook for Jira** 3. Use the search bar dropdown to quickly find or select an existing feature flag or experiment 4. Click the feature flag or experiment to link it to the issue Now your Jira issue displays essential information from GrowthBook right within your workflow. ## Viewing Flag and Experiment Details in Jira Once linked, Jira will display key details directly within your issue: ![Feature status in Jira](/images/integrations/jira/jira-feature-status.jpg) ### Feature Flags: - [Status](/features/publishing-and-approval-flows#approval-flows) - Last published/updated date - Creator - Associated experiments (if applicable) - Direct link to GrowthBook ![Experiment status in Jira](/images/integrations/jira/jira-experiment-status.jpg) ### Experiments: - [Experiment status](/app/experiment-decisions#experiment-statuses) - Phase, duration and days remaining (if running) - Linked feature flags (if applicable) - Experiment type (URL Redirect, Feature Flag, Visual Editor) - Direct link to GrowthBook ## Removing or Changing Flags or Experiments To update a flag or experiment: 1. Open the Jira issue containing the linked flag or experiment 2. Click **Replace Linked Feature** or **Replace Linked Experiment** to update the linked flag or experiment 3. Repeat the steps above to link a different flag or experiment To remove a flag or experiment: 1. Open the Jira issue containing the linked flag or experiment 2. Locate the GrowthBook panel and click the ellipsis (…) next to the linked item 3. Select **Remove** --- ## Official GrowthBook MCP Server import MaxWidthImage from '@site/src/components/MaxWidthImage'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is a standard for integrating AI tools and agents with developer platforms like GrowthBook. This guide shows you how to connect your AI tools to GrowthBook's MCP server and which tools are available for use. Demo of the MCP Server adding a feature flag to a codebase and GrowthBook :::info It's official! Check us out on the [official MCP Registry](https://registry.modelcontextprotocol.io/?q=growthbook) ::: ## Prerequisites - **Node.js**: Check if you have it by running `node -v`. Install it from [nodejs.org](https://nodejs.org/) - **GrowthBook API Key**: To create a new key, go to **Settings** → **API Keys** → **New Secret Key**. Use this key in the next step. ## Installation We cover how to connect the GrowthBook MCP server for the most popular tools below, but the installation steps are similar for most AI tools. Configuration uses the following environment variables: | Variable Name | Status | Description | | ---------------- | -------- | ----------------------------------------------------------------------------------------------------- | | GB_API_KEY | Required | A GrowthBook API key or PAT. When using a PAT, you can only access tools that your permissions allow. | | GB_EMAIL | Required | Your email address used with GrowthBook. Used when creating feature flags and experiments. | | GB_API_URL | Optional | Your GrowthBook API URL. Defaults to `https://api.growthbook.io`. | | GB_APP_ORIGIN | Optional | Your GrowthBook app URL Defaults to `https://app.growthbook.io`. | | GB_HTTP_HEADER\* | Optional | Custom HTTP headers included in all API requests. See [Custom Headers](#custom-headers) below. | :::tip On GrowthBook Cloud? You only need to configure `GB_API_KEY` and `GB_EMAIL`. ::: ### Cursor 1. Open **Cursor Settings** → **MCP** 2. Click **Add new global MCP server** 3. Add an entry for the GrowthBook MCP, following the pattern below: ```json { "mcpServers": { "growthbook": { "command": "npx", "args": ["-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "YOUR_API_URL", "GB_APP_ORIGIN": "YOUR_APP_ORIGIN", "GB_EMAIL": "YOUR_EMAIL" } } } } ``` ```json { "mcpServers": { "growthbook": { "command": "cmd", "args": ["/c", "npx", "-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "YOUR_API_URL", "GB_APP_ORIGIN": "YOUR_APP_ORIGIN", "GB_EMAIL": "YOUR_EMAIL" } } } } ``` ```json { "mcpServers": { "growthbook": { "command": "wsl", "args": ["npx", "-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "YOUR_API_URL", "GB_APP_ORIGIN": "YOUR_APP_ORIGIN", "GB_EMAIL": "YOUR_EMAIL" } } } } ``` ```json { "mcpServers": { "growthbook": { "command": "npx", "args": ["-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "YOUR_API_URL", "GB_APP_ORIGIN": "YOUR_APP_ORIGIN", "GB_EMAIL": "YOUR_EMAIL" } } } } ``` 3. Save the settings. You should now see a green active status after the server successfully connects! ![GrowthBook MCP in Cursor](/images/integrations/mcp/mcp-cursor.png) ### VS Code 1. Open **User Settings (JSON)** 2. Add an MCP entry: ```json { "servers": { "growthbook": { "command": "npx", "args": ["-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "YOUR_API_URL", "GB_APP_ORIGIN": "YOUR_APP_ORIGIN", "GB_EMAIL": "YOUR_EMAIL" } } } } ``` ```json { "servers": { "growthbook": { "command": "cmd", "args": ["/c", "npx", "-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "YOUR_API_URL", "GB_APP_ORIGIN": "YOUR_APP_ORIGIN", "GB_EMAIL": "YOUR_EMAIL" } } } } ``` ```json { "servers": { "growthbook": { "command": "wsl", "args": ["npx", "-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "YOUR_API_URL", "GB_APP_ORIGIN": "YOUR_APP_ORIGIN", "GB_EMAIL": "YOUR_EMAIL" } } } } ``` ```json { "servers": { "growthbook": { "command": "npx", "args": ["-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "YOUR_API_URL", "GB_APP_ORIGIN": "YOUR_APP_ORIGIN", "GB_EMAIL": "YOUR_EMAIL" } } } } ``` 3. Save your settings. In your file, a **Start** button will appear. Click it to the start the server. In CoPilot Chat, you'll see a tool icon, indicating the server is connected successfully. GrowthBook MCP is now ready to use in VS Code. ![GrowthBook MCP in VS Code Copilot](/images/integrations/mcp/mcp-vscode.png) ### Claude Code Claude Code uses command-line commands to add MCP servers. Run the following command in your terminal: ```bash claude mcp add growthbook --transport stdio --env GB_API_KEY=YOUR_API_KEY --env GB_API_URL=YOUR_API_URL --env GB_APP_ORIGIN=YOUR_APP_ORIGIN --env GB_EMAIL=YOUR_EMAIL -- npx -y @growthbook/mcp@latest ``` :::tip Command structure All options (`--transport`, `--env`, etc.) must come **before** the server name. The `--` (double dash) separates the server name from the command and arguments that get passed to the MCP server. ::: You can verify the server was added by running: ```bash claude mcp get growthbook ``` To see all configured MCP servers, use: ```bash claude mcp list ``` Within Claude Code, you can check server status using the `/mcp` command. ### Claude Desktop We have an official extension in the Connectors Directory. 1. Go to **Settings** → **Extensions** → **Browse Extensions** 2. Select **Desktop Extensions** and search for "GrowthBook" 3. Follow the prompts to install and configure the extension 4. That's it! :::tip Node not required Claude Desktop ships with its own version of Node, which means it's not required here ::: ## Custom Headers If your GrowthBook instance is behind a reverse proxy or requires additional authentication headers (e.g. Cloudflare Access), you can pass custom HTTP headers via environment variables using the `GB_HTTP_HEADER_*` prefix. Each environment variable following this pattern is automatically converted to a proper HTTP header and included in every API request the MCP server makes. ### Naming Convention The variable name after the `GB_HTTP_HEADER_` prefix is converted from `UPPER_SNAKE_CASE` to `Title-Case-With-Dashes`: | Environment Variable | HTTP Header Sent | | -------------------------------- | ----------------- | | `GB_HTTP_HEADER_X_TENANT_ID` | `X-Tenant-ID` | | `GB_HTTP_HEADER_CF_ACCESS_TOKEN` | `Cf-Access-Token` | | `GB_HTTP_HEADER_X_CUSTOM_AUTH` | `X-Custom-Auth` | :::info Header precedence Custom headers cannot override the `Authorization` or `Content-Type` headers, which are always set by the MCP server. ::: ### Example: Cloudflare Access If your self-hosted GrowthBook is protected by [Cloudflare Access](https://developers.cloudflare.com/cloudflare-one/policies/access/), add the service token headers: ```json { "mcpServers": { "growthbook": { "command": "npx", "args": ["-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "https://growthbook.internal.example.com", "GB_APP_ORIGIN": "https://growthbook.internal.example.com", "GB_EMAIL": "YOUR_EMAIL", "GB_HTTP_HEADER_CF_ACCESS_CLIENT_ID": "YOUR_CF_CLIENT_ID", "GB_HTTP_HEADER_CF_ACCESS_CLIENT_SECRET": "YOUR_CF_CLIENT_SECRET" } } } } ``` ```json { "mcpServers": { "growthbook": { "command": "cmd", "args": ["/c", "npx", "-y", "@growthbook/mcp@latest"], "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_API_URL": "https://growthbook.internal.example.com", "GB_APP_ORIGIN": "https://growthbook.internal.example.com", "GB_EMAIL": "YOUR_EMAIL", "GB_HTTP_HEADER_CF_ACCESS_CLIENT_ID": "YOUR_CF_CLIENT_ID", "GB_HTTP_HEADER_CF_ACCESS_CLIENT_SECRET": "YOUR_CF_CLIENT_SECRET" } } } } ``` For Claude Code, append the custom headers as additional `--env` flags: ```bash claude mcp add growthbook --transport stdio \ --env GB_API_KEY=YOUR_API_KEY \ --env GB_API_URL=https://growthbook.internal.example.com \ --env GB_APP_ORIGIN=https://growthbook.internal.example.com \ --env GB_EMAIL=YOUR_EMAIL \ --env GB_HTTP_HEADER_CF_ACCESS_CLIENT_ID=YOUR_CF_CLIENT_ID \ --env GB_HTTP_HEADER_CF_ACCESS_CLIENT_SECRET=YOUR_CF_CLIENT_SECRET \ -- npx -y @growthbook/mcp@latest ``` ### Example: Multi-Tenant Proxy If your API gateway requires a tenant identifier: ```json "env": { "GB_API_KEY": "YOUR_API_KEY", "GB_EMAIL": "YOUR_EMAIL", "GB_HTTP_HEADER_X_TENANT_ID": "your-tenant-id" } ``` You can add as many `GB_HTTP_HEADER_*` variables as needed — each one becomes a separate header on every request. ## Tools ### Feature Flags - `create_feature_flag`: Create, add, or wrap an element with a feature flag. Accepts key, type, default value, and other metadata. - `get_feature_flags`: List feature flags in your GrowthBook instance. Include a feature ID to retrieve a single flag. - `get_stale_feature_flags`: Given a list of feature flag IDs, checks whether each one is stale and returns cleanup guidance including replacement values and SDK search patterns. - `create_force_rule`: Add a targeting condition (e.g., `country = CA`) to an existing flag. - `generate_flag_types`: Generate TypeScript types for feature flags. ### Experiments - `get_experiments`: List experiments in GrowthBook. Include an experiment ID to return data about a single experiment. - This tool has 3 modes: 1. `metadata`: The default response from the GrowthBook API, including basic information about experiments. 2. `full`: Metadata about experiments plus results. Warning: This returns a large payload, which fills up the context window quickly. 3. `summary`: Presents a high-level summary of experiments and their results, combining experiment metadata, results, and metrics. - `get_attributes`: List user attributes tracked in GrowthBook (useful for targeting). - `create_experiment`: Creates an A/B test. Optionally, provide a feature ID to link the experiment to a feature flag. Note: this tool calls `get_defaults`, which provides a sampling of previous experiments to help the AI agent create the experiment correctly. For better performance, these defaults are saved to the local file system and periodically updated. - `get_defaults`: Get default values for experiments including hypothesis, description, datasource, and assignment query. (Runs automatically when the create experiment tool is called.) - `create_defaults`: Set custom default values for experiments that will be used when creating new experiments. - `clear_user_defaults`: Clear user-defined defaults and revert to automatic defaults. ### Environments - `get_environments`: List all environments (e.g., production, staging) configured in GrowthBook. ### Projects - `get_projects`: List all projects in your GrowthBook instance. ### SDK Connections - `get_sdk_connections`: List SDK connections (how GrowthBook connects to your apps). - `create_sdk_connection`: Create a new SDK connection for your app, specifying language and environment. ### Metrics - `get_metrics`: List fact and legacy metrics (useful for better understanding experiments). Include a metric ID to return details about a specific metric. ### Documentation Search - `search_growthbook_docs`: Search the GrowthBook documentation for information on how to use a feature, by keyword or question. ## Example Prompts Here are some prompts to try with your AI assistant once the MCP server is connected: ### Feature Flags - "Create a boolean feature flag called `new-checkout-flow` that defaults to `false`" - "Show me the details and rules for the `dark-mode` feature flag" - "Add a force rule to `new-checkout-flow` that enables it for users where `country` is `US`" - "Check if these flags are stale and can be cleaned up: `old-banner`, `beta-pricing`, `temp-fix`" - "Generate TypeScript types for all my feature flags" ### Experiments - "Summarize the results of my last 5 experiments" - "Create an A/B test on the `new-checkout-flow` flag with a control (`false`) and treatment (`true`)" - "Show me the full results for experiment `exp_abc123`, including metric breakdowns" ### General - "What SDK connections do I have set up?" - "Search the GrowthBook docs for how to set up a fact metric" - "List all my projects and environments" ## Video Tutorial Watch this video for a complete walkthrough of setting up and using the GrowthBook MCP Server: ## Next steps Now that you're connected, start using GrowthBook's MCP tools directly from your editor to: - Add new flags to your code and GrowthBook - Generate type definitions for your flags for [typesafe feature flagging](https://github.com/growthbook/examples/tree/main/vanilla-typescript) - Create new feature-flag experiements - Search the GrowthBooks docs without leaving the editor - Check out the [GrowthBook MCP Server code on GitHub](https://github.com/growthbook/growthbook-mcp) [Join our community Slack for additional tips and tricks](https://join.slack.com/t/growthbookusers/shared_invite/zt-2xw8fu279-Y~hwnfCEf7WrEI9qScHURQ) --- ## Configuring SCIM for GrowthBook import CommercialFeature from '@site/src/components/CommercialFeature'; # SCIM Integration for Enterprise Organizations SCIM, or [System for Cross-domain Identity Management](https://scim.cloud/), is the standard for managing users and groups across multiple applications. With SCIM, you can automate the provisioning and deprovisioning of users in your GrowthBook account through your identity provider. GrowthBook's SCIM integration currently offers the following features: - User provisioning - User deprovisioning - Group Push When a user is provisioned, they are added to your GrowthBook organization with your organization's default role. After provisioning, admin users can adjust their roles and permissions via the GrowthBook application as needed. It's important to note that if a user is provisioned through SCIM, they can only be deprovisioned through your identity provider. GrowthBook does support the ability to optionally [define a role when provisioning a user](#okta-how-to-define-user-roles-when-provisioning). Project-level permissions are not supported when provisioning a user via SCIM. That must be done via the GrowthBook application. When a group is added to "Push Groups" in your Okta SCIM application, groups and their members will be synced with GrowthBook via a corresponding [Team](/account/user-permissions). Please note that only members that have been provisioned into the GrowthBook app will be added to the Team in GrowthBook. For example, if you have a push group with members A, B, and C and only A and B are assigned to GrowthBook on the "Assignments" tab, the corresponding team on GrowthBook will only include members A and B. Just like how user provisioning works, new Teams will be set up with your organization's default role. You'll need to adjust the permissions for the Team in GrowthBook. Group removal or membership changes will need to be done through your identity provider. :::note We are actively working on adding support for additional identity providers. ::: ## Configuring SCIM Integration ### Okta Setup 1. Verify that your GrowthBook organization is on an enterprise plan with SSO enabled. 2. Log in to your Okta account and go to the Applications page. Select "Browse App Catalog," then search for "SCIM 2.0 Test App (OAuth Bearer Token)." Click "Add Integration" to add the app to your Okta account. ![Add GrowthBook SCIM integration to Okta](../../static/images/integrations/scim/GrowthBook-SCIM-Add-Okta-Integration.png) 3. Once the app is added, you can change its name, for example, to "GrowthBook SCIM." Click "Next." ![Add name to GrowthBook SCIM integration](../../static/images/integrations/scim/GrowthBook-SCIM-Add-Name.png) 4. On the next page, you don't need to modify any settings. Simply click "Done." 5. With the application created, click on the "Provisioning" tab and select "Configure API Integration." ![Enable API integration for GrowthBook SCIM](../../static/images/integrations/scim/GrowthBook-SCIM-Enable-API-Integration.png) 6. Now you can enter the Base URL and your OAUTH Bearer Token. - For GrowthBook Cloud users, the Base URL is `https://api.growthbook.io/scim/v2`. If you're self-hosting GrowthBook, the Base URL will be `{YOUR_API_HOST}/scim/v2`. - You can obtain your OAuth Bearer Token by creating a new Secret API Key with an `Admin` role. To do this, go to your GrowthBook account, and in the left navigation, select "Settings → API Keys." We recommend creating a dedicated API key exclusively for your SCIM integration. :::info Allowlist Okta's IP addresses If self-hosting GrowthBook on a VPN or behind a firewall, allowlist [Okta's IP addresses](https://support.okta.com/help/s/article/list-of-ip-addresses-that-should-be-allowlisted-for-inbound-traffic?language=en_US) to ensure SCIM API calls can hit GrowthBook's SCIM server. ::: 7. After adding your credentials, click "Test API Credentials" to ensure they are valid. If they pass the test, click "Save." ![](../../static/images/integrations/scim/GrowthBook-SCIM-Add-Credentials.png) 8. Next, click on the "To App" tab and select "Edit" to enable "Create Users" and "Deactivate Users." Once enabled, click "Save." ![](../../static/images/integrations/scim/GrowthBook-SCIM-Configure-Options.png) 9. Congratulations! Your application is now set up. You can navigate to the "Assignments" tab and assign people to GrowthBook and to the "Push Groups" tab to sync groups with GrowthBook Teams. #### Okta: How to Define User Roles When Provisioning 1. First, ensure that you've followed the general setup instructions above. 2. In Okta, navigate to the Application you created for GrowthBook SCIM, and click on the "Provisioning" tab, and scroll to the "Attribute Mappings" section, before clicking "Go to Profile Editor". 3. Then, you'll click "Add Attribute" to create a new attribute. with the following details. - Data type: string - Display name: GrowthBook Role - Variable name: growthbookRole - External name: growthbookRole - External namespace: urn:ietf:params:scim:schemas:core:2.0:User - Check the box to define an enumerated list of values - Here, you need to add the following enum Options - Display name: Read only - Value: readonly - Display name: Collaborator - Value: collaborator - Display name: Engineer - Value: engineer - Display name: Analyst - Value: analyst - Display name: Experimenter - Value: experimenter - Display name: Admin - Value: admin ![](../../static/images/integrations/scim/GrowthBook-SCIM-Define-Role-Attribute.png) :::info Exact capitalization required It's important each of these values are entered exactly as shown above, including capitalization. If there is a descrepancy, the user's role will fall back to your organizations default role. ::: 4. Once complete, when you provision a user, you can select their role from the dropdown, and that will be applied to the user in GrowthBook. If you're not sure which role a user should have, you can [view GrowthBook role permissions](/account/user-permissions). ![](../../static/images/integrations/scim/GrowthBook-SCIM-Define-User-Role.png) ### Azure AD/Microsoft Entra ID Setup The setup directions below are shown using the Azure Portal (e.g., portal.azure.com); however, the steps are the same if you're setting this up via the Microsoft Entra Admin Panel (e.g., entra.microsoft.com). 1. Verify that your GrowthBook organization is on an enterprise plan with SSO enabled ![Microsoft Azure dashboard, showing the Manage/Applications page](../../static/images/integrations/azure/growthbook-entra-scim-add-enterprise-application.png) 2. Log in to your Azure account, click **Microsoft Entra ID**, and in the sidebar menu, select **Manage** → **Enterprise Applications** → **+ New Application** 3. Select **+ Create your own application**, add an app name, and select **Integrate any other application you don't find in the gallery (Non-gallery)**. ![A text input on the create your own application form in Microsoft Azure](../../static/images/integrations/azure/growthbook-entra-scim-create-application.png) 3. Once created, select the box to **Provision User Accounts** ![Microsoft Azure web app showing a Getting Started page with the Provisioning User Accounts option highlighted](../../static/images/integrations/azure/growthbook-entra-scim-setup-provisioning.png) 4. Click **Get Started** ![Microsoft Azure web app with the lifecycle management page](../../static/images/integrations/azure/growthbook-entra-scim-provisioning-empty-state.png) 5. Change the Provisioning Mode to **Automatic** before adding the Base URL and your OAUTH Bearer Token ![Microsoft Azure web app on the provisioning mode page](../../static/images/integrations/azure/growthbook-entra-scim-add-credentials.png) - For GrowthBook Cloud users, the Base URL is `https://api.growthbook.io/scim/v2`. If you're self-hosting, the Base URL is `{YOUR_API_HOST}/scim/v2`. - Obtain your OAuth Bearer Token by creating a new Secret API Key with an `Admin` role. To do this, open GrowthBook and go to **Settings** → **API Keys**. We recommend creating a dedicated API key for your SCIM integration. :::info Allowlist Azure AD's/Entra ID's IP addresses If self-hosting GrowthBook on a VPN or behind a firewall, allowlist Azure AD's/Entra ID's IP addresses to ensure the SCIM API calls can hit GrowthBook's SCIM server. ::: 7. After adding your credentials, click **Test API Credentials** to ensure they're valid. If they pass the test, click **Save**. 8. Once saved, enable/disable syncing of groups and users on the **Mappings** page. The two screenshots below show the necessary mappings. These must match exactly! #### User Mapping Customizing the attribute mapping is a critical step that enables the mapping of GrowthBook attributes to the correct Azure AD / Entra ID attributes. Below are the required attributes and how they need to be mapped for users. | GrowthBook Attributes | Azure AD / Entra ID Attributes | Matching precedence | | :-------------------: | :---------------------------------------------------------: | :-----------------: | | userName | userPrincipalName | 1 | | active | Switch([IsSoftDeleted], , "False", "True", "True", "False") | - | | displayName | displayName | - | | externalID | objectId | - | ![Microsoft Azure web app on the attribute mappings page](../../static/images/integrations/azure/growthbook-entra-scim-user-mapping.png) #### Group Mapping (Optional) Using groups simplifies providing users with custom permissions. When a user is added to a group and the group is provisioned to GrowthBook, we'll create a Team in GrowthBook, and all users of the team will inherit the team's permissions. See more [details on teams](/account/user-permissions#teams). If you don't plan on using `Teams`, you can disable group syncing and do not have to customize the attribute mapping. If you are planning to use groups, below are the required attributes and how they need to be mapped. | GrowthBook Attributes | Azure AD / Entra ID Attributes | Matching precedence | | :-------------------: | :----------------------------: | :-----------------: | | displayName | displayName | 1 | | members | members | - | ![Microsoft Azure web app on the attribute mappings page, extended](../../static/images/integrations/azure/growthbook-entra-scim-group-mapping.png) :::info Syncing groups is optional When a group is synced with GrowthBook, it creates a team in your GrowthBook organization and adds any members in the group to the team. This is handy because any role you assign to a team will also be inherited by its members. With Azure AD/Entra ID, all group members are synced to GrowthBook. ::: ![Microsoft Azure on the SCIM overview page](../../static/images/integrations/azure/growthbook-entra-scim-start-provisioning.png) 9. Add users and groups. Then, click **Start Provisioning**. ## FAQs **What features are supported with SCIM?** GrowthBook supports provisioning and deprovisioning users and groups. **What identity providers are supported?** GrowthBook officially supports Okta and Azure AD/Microsoft Entra ID. Given the standarized nature of SCIM, it's possible other identity providers will work, but we do not have official guides at this time. **What happens if I deprovision a user in my identity provider?** If a user is deprovisioned in your identity provider, they will be removed from GrowthBook. If they are re-provisioned, they will be added back to GrowthBook, and their role will reset to the organization's default role. **All the users from my group aren't being synced with my GrowthBook team. What's happening?** If you use Okta and notice that some users that you expect to be in your GrowthBook team from your identity provider are not being added in GrowthBook, double check that those users are assigned to the GrowthBook SCIM application in your identity provider. - **Okta**: Only users that are both assigned and within the group are synced to the corresponding team. - **Azure AD/Entra ID**: The user only needs to be in a group assigned to the application to be provisioned. **What if I already have users in GrowthBook?** Existing users in GrowthBook will not be affected by SCIM. You can continue to manage them through the GrowthBook application as usual. If you wish to transition them to be managed by your identity provider, you can provision them through your identity provider. As long as the email matches, the existing GrowthBook user will be converted to be managed by your identity provider. **Does GrowthBook follow SCIM 1.1 or 2.0 Protocol?** GrowthBook follows the SCIM 2.0 protocol. **What happens if I provision a user with a role that doesn't exist?** GrowthBook will fallback to your organization's default role. **Can I provision a user with project-specific permissions?** No, at this time, you can only provision a user's global role. Project-specific permissions must be managed through the GrowthBook application. **Can I change a user's global role after they've been provisioned?** Yes, you can change a user's global role through the GrowthBook application or via Okta. Please note that if you change a user's global role through the GrowthBook Application, it will then be out-of-date in Okta. If you then update the user in Okta, it will update the user's global role. **I'm changing a user's role in Okta, but it's not changing in GrowthBook.** For the user's role to be updated in GrowthBook, you must update your Application provisioning to support "updates." The only property GrowthBook supports updating is `growthbookRole`. **Why can't I change a user's name via my identity provider?** In GrowthBook, a user can belong to multiple organizations, so no single organization can update a user's name, as it would impact all organizations the user is part of. **Can I define a user's `growthbookRole` via Azure AD/Microsoft Entra ID?** It's possible to define `growthbookRole` by adding a custom attribute and mapping it accordingly, but we don't yet offer any additional documentation. **I'm using Microsoft Entra ID/Azure AD and I'm unable to sync groups.** Please ensure that your Microsoft Entra ID/Azure AD plan supports group syncing. **How do I change a user's email address once provisioned?** It's not possible to update a user's email after they're provisioned. Instead, follow these steps: - Deprovision the user - Update their email - Reprovision the user This process creates a net-new user in GrowthBook with the updated email address. --- ## Shopify Integration import CommercialFeature from '@site/src/components/CommercialFeature'; # Shopify Integration Unleash the power of experimentation with GrowthBook to supercharge your Shopify store—no coding skills required! ## Let's Get Started ### Step 1: Create a GrowthBook SDK Connection To connect your GrowthBook account to Shopify, you'll need to create a new SDK Connection and select the `Shopify` option. Before saving, confirm that you've enabled the toggles for `Include visual experiments in endpoint's response?` and `Include draft experiments`. ![](../../static/images/integrations/shopify/shopify-sdk-connection.png) ### Step 2: Add GrowthBook to Your Shopify Theme Once the SDK Connection is created, you should see a code snippet that you need to add to your store. Log into your Shopify account, navigate to your online store, and edit your store's code. ![](../../static/images/integrations/shopify/growthbook-shopify-edit-code.png) Once you're in the code editor, locate the `theme.liquid` file and add the snippet exactly as shown just before the closing `` tag. Here's an example of how it should look: ![](../../static/images/integrations/shopify/shopify-theme-editor.png) :::note Tracking experiment views When a user views an experiment, this script will fire an event that tracks which variation the user saw. If you have Google Analytics 4 (GA4) or Segment installed on your store, you don't have to do any additional configuration. If, however, you need to use a different analytics provider, you can follow our guide [here](/lib/script-tag). ::: Once added, click "Save" and navigate back to GrowthBook. ### Step 3: Install the GrowthBook Visual Editor Install the [GrowthBook Visual Editor](https://chromewebstore.google.com/detail/growthbook-visual-editor/nbomejknbpkcpjdagefhichaajpoempk) from the Chrome Web Store, then open the side panel and click **Connect with GrowthBook**. See [Install and connect](/app/visual/install-and-connect) for the full setup. ### Step 4: Create a GrowthBook Visual Editor Experiment :::tip Questions about the Visual Editor? If you have questions about how to use the Visual Editor, [check out our guide here](/app/visual). ::: Now that GrowthBook is installed on your Shopify store and you've installed the browser extension, you can create a new experiment using the Visual Editor. First, navigate to "Experiments" on the left-hand navigation menu and then click "Create Experiment", before selecting "Design a New Experiment" and following the on-screen prompts. Once you've created your experiment, select "Visual Editor" as the editor type and enter the URL of your Shopify store. ![](../../static/images/integrations/shopify/growthbook-shopify-open-visual-editor.png) You'll then be redirected to your store, where you can use the Visual Editor to update your store's content. --- ## Slack integration import ExternalLink from '@site/src/components/ExternalLink' # Slack integration The GrowthBook Slack integration allows you to receive alerts for the events that you care about in a Slack channel of your choosing. ## Creating and configuring an app in Slack ### Create a Slack app The first step to setting up the GrowthBook Slack integration is to create a new app under your workspace's owned apps. You will need administrative privileges for your workspace in order to be able to manage apps. Navigate to your workspace’s apps and choose to create a new app. You can get to that page directly [here](https://api.slack.com/apps?new_app=1). ![](../../static/images/integrations/slack/slack-create-app.png) Name the app and choose the workspace. In this case we are created the app named GrowthBook Alerts and are picking the GrowthBook Dev workspace. ![](../../static/images/integrations/slack/slack-name-app-choose-workspace.png) ### Create an Incoming Webhook We will be alerting via Slack's incoming webhooks functionality. Under the **Basic Information** tab of your app, click on **Incoming Webhooks**. ![](../../static/images/integrations/slack/slack-incoming-webhooks.png) ### Subscribe a channel to alerts Click on the button at the bottom of the **Incoming Webhooks** page that says **Add New Webhook to Workspace**. You will be asked to install the app into your workspace. Choose the desired channel you'd like to receive notifications in. ![](../../static/images/integrations/slack/slack-install-flow.png) Once you've completed this flow, you will see a URL available for you to copy. Save this value for later. ![](../../static/images/integrations/slack/slack-copy-webhook-url.png) ## Add the Slack integration to GrowthBook Next step is to login to the GrowthBook app and visit the **Webhooks** tab under **Settings**, also available [here](https://app.growthbook.io/settings/webhooks). You will need privileges to manage webhooks in order for this menu item to be available. Click the **Create an Event Webhook** button. You should see a modal pop up with some fields for configuring the Slack notification webhook. ![](../../static/images/webhooks/slack/new-webhook.png) Then, configure the following: - **Name**: The name of the integration. In case you have multiple integrations, this can help you tell them apart. This will also show in the contextual text alongside the alerts. - **Endpoint URL**: Copy and paste the webhook URL provided by your slack app. - **Payload**: Select the `Slack` payload. - **Event filters**: You can optionally filter by events you care about. For example, if you only care about when features are deleted, you can choose `feature.deleted` from the list. If you care about all events, leave this blank. - **Environment filters**: You can optionally choose to filter by environment. For example, if you only want to hear about events that are for the production environment, you can choose `production` from the list. For all environments, leave this blank. - **Project filters**: You can optionally choose to filter by project. For example, if you have a project named "Onboarding V2" and you only want to alert for that project, you can choose that project from the projects list. For all projects, leave this blank. - **Tag filters**: You can optionally choose to filter by tag. For all events regardless of tag, leave this blank. After configuring all the fields, press **Create** to save your new Slack notification webhook. You can also edit these fields at any point if you make a mistake. ![](../../static/images/webhooks/slack/edit-webhook.png) ### Adding more alerts If you'd like to be alerted in another Slack channel, you can add another Incoming Webhook in Slack. Then, you can create new integrations in the GrowthBook dashboard, specifying all of the same information except adding your new webhook URL. ## Testing your alerts You are now ready to test your alerts. First, you can hit the `Test` button on the webhook settings page. This should trigger a test notification. Next, perform one of the actions you're watching if you've added Event filters. If you haven't added any event filters, the quickest way to test it's working is to either create a new test feature (then delete it if it's not needed), or toggle an environment on or off for an existing feature. --- ## Vercel Native Integration GrowthBook is now available as a Native Integration in Vercel, allowing you to manage feature flags and experiments directly from your Vercel dashboard. This integration also works with Edge Config, enabling you to use GrowthBook's SDKs with near-zero latency by serving flags from the edge without API calls. ## What's Included - View all of your feature flags and experiments in Vercel with links to GrowthBook for more details. - Login to GrowthBook directly from Vercel without needing to manage separate credentials. - Sync your feature flags to Vercel Edge Config to bootstrap GrowthBook's SDKs with near-zero latency. - Install into multiple Vercel projects - each one will get their own corresponding project in GrowthBook. - Seamlessly integrate with Vercel's Flags SDK with zero-configuration - ideal for Next.js applications. ## Installation Installation is done entirely through the Vercel dashboard: 1. Go to your Vercel dashboard and navigate to the **Integrations** section. 2. Click on **Browse Marketplace**. 3. Search for "GrowthBook" and select the GrowthBook integration. 4. Click **Install** and follow the prompts. :::warning This integration creates a brand new GrowthBook organization and cannot be used to connect to an existing GrowthBook organization. ::: ## Billing When configuring the integration, you can choose our free Starter plan or Pro plan. When using the Pro plan, you will be billed directly through Vercel. You can manage your billing info, view current usage, download past invoices, and cancel your subscription directly from the Vercel dashboard. The Pro plan includes a per-seat fee. You are only charged for members on your team who log into the GrowthBook application. The user who installs the integration will be your first seat. Viewing flags and experiments directly within the Vercel dashboard is always 100% free. If you no longer need a seat (e.g. someone leaves your organization), you can remove them from GrowthBook's member settings page and you'll no longer be charged for their seat. ## User Management Any user in your Vercel team can log into GrowthBook by clicking the "Open in Provider" button or clicking into a flag or experiment within Vercel. This will automatically create a GrowthBook user for them if they don't already have one and grant them the necessary permissions to manage feature flags and experiments. Admins within GrowthBook can manage user roles and permissions. Only the first user who installs the integration will be an admin by default and you must always have at least one admin user in GrowthBook. ## Edge Config and SDK Integration You can optionally enable Edge Config syncing during the installation process. When enabled, GrowthBook will automatically sync your feature flags to Vercel Edge Config. Edge Config is Vercel's global data store that provides ultra-low latency access to configuration data at the edge. By syncing your feature flags to Edge Config, your applications can evaluate flags without making API calls to GrowthBook's servers. You can use this Edge Config data with any of our [SDKs](/lib), but the most seamless integration is with Next.js using Vercel's Flags SDK. View the docs for the [Next.js SDK](/lib/nextjs) for more details. ## Next Steps Once your integration is installed: 1. **Create your first feature flag**: Click into your GrowthBook organization from the Vercel dashboard 2. **Set up your SDK**: If you enabled Edge Config, check out our [Next.js SDK guide](/lib/nextjs) for the fastest setup 3. **Invite your team**: Add team members who will need to manage flags and experiments 4. **Start experimenting**: Create your first A/B test and start collecting data For additional help, [join GrowthBook's official Slack community](https://join.slack.com/t/growthbookusers/shared_invite/zt-2xw8fu279-Y~hwnfCEf7WrEI9qScHURQ). --- ## Webflow Integration # Webflow Integration Unleash the power of experimentation with GrowthBook to supercharge your Webflow site—no coding skills required! :::info Note This guide walks you through creating experiments using the GrowthBook Visual Editor, which requires a `Pro` subscription. [Learn More](https://www.growthbook.io/pricing). ::: ## Let's Get Started ### Step 1: Create a GrowthBook SDK Connection To connect your GrowthBook account to Webflow, you'll need to create a new SDK Connection and select the `Webflow` option. Before saving, confirm that you've enabled the toggles for `Include visual experiments in endpoint's response?` and `Include draft experiments`. ![](../../static/images/integrations/webflow/webflow-sdk-connection.png) ### Step 2: Add GrowthBook to Your Webflow Site Once the SDK Connection is created, you should see a code snippet that you need to add to your site. In a new window, log into your Webflow account, navigate to your site's settings, and select the `Custom Code` tab. Find the "Head Code" section, and paste in the code snippet from GrowthBook. It should look something like this: ![](../../static/images/integrations/webflow/growthbook-webflow-edit-code.png) :::info Tracking experiment views When a user views an experiment, the code above will fire an event that tracks which variation the user saw. If you have Google Analytics 4 (GA4) or Segment installed on your site, you don't have to do any additional configuration. If, however, you need to use a different analytics provider, you can follow our guide [here](/lib/script-tag). ::: Once added, save and publish the changes and navigate back to GrowthBook. ### Step 3: Install the GrowthBook Visual Editor Install the [GrowthBook Visual Editor](https://chromewebstore.google.com/detail/growthbook-visual-editor/nbomejknbpkcpjdagefhichaajpoempk) from the Chrome Web Store, then open the side panel and click **Connect with GrowthBook**. See [Install and connect](/app/visual/install-and-connect) for the full setup. ### Step 4: Create a GrowthBook Visual Editor Experiment :::info Questions about the Visual Editor? If you have questions about how to use the Visual Editor, [check out our guide here](/app/visual). ::: Now that GrowthBook is installed on your Webflow site, you can create a new experiment using the Visual Editor. First, navigate to "Experiments" on the left-hand navigation menu and then click "Create Experiment", before selecting "Design a New Experiment" and following the on-screen prompts. Once you've created your experiment, select "Visual Editor" as the editor type and enter the URL of your Webflow site. ![](../../static/images/integrations/webflow/growthbook-webflow-visual-editor.png) You'll then be redirected to your site, where you can use the Visual Editor to update your site's content. --- ## Wordpress Integration # Wordpress Integration Unleash the power of experimentation with GrowthBook to supercharge your Wordpress site—no coding skills required! :::info Note This guide walks you through creating experiments using the GrowthBook Visual Editor, which requires a `Pro` subscription. [Learn More](https://www.growthbook.io/pricing). ::: ## Let's Get Started ### Step 1: Create a GrowthBook SDK Connection To connect your GrowthBook account to Wordpress, you'll need to create a new SDK Connection and select the `Wordpress` option. Before saving, confirm that you've enabled the toggles for `Include visual experiments in endpoint's response?` and `Include draft experiments`. ![](../../static/images/integrations/wordpress/wordpress-sdk-connection.png) ### Step 2: Add GrowthBook to Your Wordpress Site Once the SDK Connection is created, you should see a code snippet that you need to add to your site. The easiest way to do this is with the popular [Insert Headers and Footers Plugin](https://wordpress.org/plugins/.insert-headers-and-footers/) by WPCode. Install and activate this plugin if you don't have it already. On the left-hand side of your dashboard, navigate to Code Snippets → Header & Footer. Insert the SDK Connection code snippet into your site's Head: ![Add code to header](../../static/images/integrations/wordpress/add-code-to-header.png) Here's an example of how it should look: ```html ``` :::info Tracking experiment views When a user views an experiment, this script will fire an event that tracks which variation the user saw. If you have Google Analytics 4 (GA4) or Segment installed on your site, you don't have to do any additional configuration. If, however, you need to use a different analytics provider, you can follow our guide [here](/lib/script-tag). ::: Once added, navigate back to GrowthBook. ### Step 3: Install the GrowthBook Visual Editor Install the [GrowthBook Visual Editor](https://chromewebstore.google.com/detail/growthbook-visual-editor/nbomejknbpkcpjdagefhichaajpoempk) from the Chrome Web Store, then open the side panel and click **Connect with GrowthBook**. See [Install and connect](/app/visual/install-and-connect) for the full setup. ### Step 4: Create a GrowthBook Visual Editor Experiment :::info Questions about the Visual Editor? If you have questions about how to use the Visual Editor, [check out our guide here](/app/visual). ::: Now that GrowthBook is installed on your Wordpress site and you've installed the browser extension, you can create a new experiment using the Visual Editor. First, navigate to "Experiments" on the left-hand navigation menu and then click "Create Experiment", before selecting "Design a New Experiment" and following the on-screen prompts. Once you've created your experiment, Add a Visual Editor change and enter the URL of your Wordpress site. You'll then be redirected to your site, where you can use the Visual Editor to make changes. --- ## A/A Testing in GrowthBook ## What is an A/A Test? A/A tests are a form of A/B test where there is no difference between variations (each "variation" is labeled as "A"). They are used to test whether your experimentation set up and methodology is working as expected. Users who are placed into your A/A test receive the same user experience regardless of which variation they view. Thus, you'll expect to see almost identical results for each "variation" since all users are viewing the same thing. This helps to ensure that you've integrated the GrowthBook SDK correctly, that traffic is flowing as expected, and that your Metrics are defined properly, all _before_ you begin experimenting on features that matter! Because all users get the same user experience, any variation in traffic flows or Metric values can be attributed either to errors in your experiment or SDK setup, or to random chance. Metric variations in A/A tests are covered in more detail below. ## When should I run an A/A test in GrowthBook? There are two scenarios in which it is a good idea to run an A/A test: **You've set up a new SDK connection.** Once your SDK is connected and you configure the trackingCallback method, you can validate that the experiment data is being sent correctly to your data warehouse. **There are changes to any part of the GrowthBook integration**, such as changes to your data warehouse, the tracking libraries (Google Analytics [GA4], Segment, Amplitude, etc.), or changes to the GrowthBook-related code within your application. ## How do I run an A/A test in GrowthBook? An A/A test is run just as an A/B test would be run. Follow the usual instructions for creating an experiment and ensure the "variations" are serving _the same values_. The feature for your A/A test will look something like this: ![A/A Feature Rule](/images/aa-feature-rule.png) Then, start your experiment just as you would any other experiment. Wait until it has been live for a few hours or days, depending on how much traffic you get, then take a look at the results. ## What problems can an A/A test reveal? A/A tests are a low-risk approach to help ensure that your experiment is set up correctly. ### Problem: No traffic, incorrect metric values, or SRM errors (imbalanced traffic) For these errors, check our [troubleshooting guide](/kb/experiments/troubleshooting-experiments). ### Problem: Metrics show statistically significant lifts in the A/A test The main goal of an A/A test is to ensure your experimentation system is set up correctly. One common way to test this is to confirm that all of the Metrics appear balanced across the two variations. In an ideal world, there would be no statistically significant Metric movements between the two variations, because if everything is set up correctly the two variations should have produced (nearly) identical results. In a clearly broken A/A test, you might see 3-4 Metrics all statistically significant with > 99%+ or < 1% chance to win. This is a clear sign that something is wrong with the experimentation set up. However, there are often cases where only one or two Metrics are significant, and maybe only just across your significance thresholds. Often this is not a problem and can be a false positive, especially if you added many metrics to your A/A test. Understanding why this is requires a bit of background on statistical significance and false positives in experiments. #### Understanding significance and false positives rate In experimentation, it is necessary to allow some amount of "wiggle room" when calling a result "statistically significant". In GrowthBook, our default is to state that a Metric is statistically significant if there is a 95% chance that the variation is better than the baseline (a positive result), or only a 5% chance that it is better (a negative result). This means that a significant positive result is actually better than the baseline 95% of the time; in other words, 5% of the time, it isn't actually better! We allow this 5% chance, also known as the "false positive rate", because we want to be able to declare a winner in an experiment even when we're not 100% certain. For an A/A test, this means that even if your experiment set up is perfect and everything is balanced, you still have a chance of seeing a statistically significant result due to randomness. Even if you run your A/A test for months, you can still see some imbalance just due to bad luck in the randomization. :::info False positive rate If you run an A/A test with one Metric and use GrowthBook's default 95% and 5% thresholds, you have a 10% chance of seeing a "statistically significant" result even when there is no real difference between the variations. Furthermore, the more Metrics you add to your test, the greater the probability of seeing a false positive rate! If you have 2 unrelated Metrics in your A/A test, your chance of getting at least one false positive is 19%, and with 5 unrelated Metrics, it's 41%! ::: You can lower your false positive rate by lowering the "Chance to Win Threshold" in your Organization settings, but that just means it will also take longer for actual A/B experiments to reach your significance level. So, what does this mean practically for A/A tests? - If you have an A/A test with 10 substantively different Metrics, and 7 of them are statistically significant, and the differences are large (e.g. not 96% chance to win, but 99.9+), then you can plausibly state that something is wrong and you may want to dig into your set up further. Use the general solutions in the [troubleshooting guide](/kb/experiments/troubleshooting-experiments) as hints for where to begin looking for errors. - If you have an A/A test with 10 Metrics and 1 of them is statistically significant, or even 2 of them, then this is very plausibly due to chance and likely can be ignored and deemed a successful A/A test. Furthermore, if there are no SRM errors or other signs of issues with your experiment, that could lend further credibility to these few significant results being a false positive. You can restart your A/A test with re-randomization to confirm this if you want to. You should see a similar number of metrics reaching statistical significance, and they should not always be the same ones. - Things get more complicated when you have 10 Metrics and 3-4 of them show statistically significant differences. In these cases, you should consider whether the Metrics are related to one another. If you have three "purchase" Metrics that are highly correlated with one another, then it's plausible that one "unlucky" draw caused this statistical significance and the A/A test is working fine. In this situation, you should consider restarting your A/A test to confirm this theory. If all three Metrics are quite different from one another, then it's stronger evidence that something may be wrong in your set up. ## Restarting an A/A test If you found some issues with your set up and want to re-run your A/A test, or if you had a Metrics imbalance that you are still uncertain about, you should usually restart the A/A test and run it again. You have two options: - Create an entirely new experiment. This may be the best and cleanest option since you ensure you are using a new tracking key, new set up, and are unlikely to have any issues carrying over from the old experiment. - Restart the existing experiment with a new phase. This is the easiest approach, and will likely work in most cases. You can do this as follows: Use the "Make Changes" flow in the A/A experiment page. Use this flow to "Start a New Phase" and make sure you "re-randomize traffic" to ensure that users returning to your test will be re-randomized into variations (this will re-set the seed for the experiment, causing users to get randomly re-assigned). ![New Phase](/images/new-phase.png) ![New Phase, Re-Randomize](/images/new-phase-rerandomize.png) --- ## Carryover Bias # What is Carryover Bias? In most experimentation systems, such as GrowthBook, users are randomly assigned to different variations (A, B, C, etc.) in a way that ensures that past experiments do not affect one variation more than any other. However, if an experimenter creates a new phase of an experiment without re-randomizing user assignment to variations the new phase of the experiment will be affected by **carryover bias**. This happens when restarting an experiment and when creating an analysis using the last few days or weeks of an experiment. In short, without re-randomization, the effects of the first phase of the experiment can contaminate the second phase of the experiment if any users return to the experiment in the second phase. The rest of this document provides an example of carryover bias and how to avoid it in GrowthBook. ## Example of Carryover bias Imagine an experiment with two variations (A and B) that runs for 3 days before you realize that there is a bug in variation B that is causing users to churn at a high rate. Let's call this Phase 1. To mitigate this issue, you fix the bug and now want to restart your experiment. Furthermore, you don't want to re-randomize users because you want returning users in Phase 2 to get the same variation that they saw in Phase 1. So, you restart the experiment without re-randomizing users (let's call this Phase 2). However, **this choice to not re-randomize will lead to carryover bias**! How so? Well, consider what will happen after you restart the experiment. Let's say in Phase 2 each variation gets 100 totally new users. However, we also have some users returning to the app who were exposed to the experiment in Phase 1. In variation A, there are 50 returning users and in variation B, where the bug was causing churn, there are 10 returning users. Then, your experiment has 150 users in A and 110 users in B! This imbalance is caused because the effects of first phase of the experiment is _carrying over_ to the second phase of the experiment. The best way to deal with carryover bias is to prevent it from happening in the first place, because this kind of bias can **unpredictably affect your results** and is **hard to detect**. #### The effects of carryover bias can be unpredictable Carryover bias works to affect your results in many different ways: - It could cause B to look **better** than it is; if the 10 people that returned in variation B are particularly persistent power users, then B will look good not because of the feature you're testing, but because you're getting power users bucketed into variation B. - It could cause B to look **worse** than it is because the bad taste of the buggy experience from Phase 1 could cause returning users in B to be less engaged. That "bad taste" won't be spread across variations in Phase 2 because you did not re-randomize users. #### Carryover bias can be hard to detect In the above example, the discrepancy in users in variations B and A will eventually trigger a Sample Ratio Mismatch (SRM) warning, alerting you to some issue with your experiment. However, SRM warnings will not always fire in cases where there is carryover bias, because either (1) the discrepancy is too small or the total user count is too low; or (2) the carryover bias doesn't actually come from differences in user return rates, but comes from other differences in user behavior across variations that was caused by the first phase of the experiment. ## What causes carryover bias and how can I prevent it? In short, anything that causes users in an experiment analysis to not be randomly assigned to a variation can cause bias. For carryover bias specifically, common causes are: ### 1. Creating a new "Phase" to restart an experiment without re-randomizing In this case, carryover bias is best prevented by re-randomizing users whenever you restart an experiment — either by starting an entirely new experiment or choosing to "re-randomize", which is an option we provide in GrowthBook's SDKs. We provide specific guidance on when you should re-randomize users in GrowthBook as part of our ["Make Changes" flow](/app/making-experiment-changes). ### 2. Manually restarting a new experiment with the same experiment key Similarly, if you also manually restart an experiment with the same experiment key, the randomization from the previous experiment will carry over to the new experiment. It's best in these cases to change the experiment key to ensure users are re-randomized into groups. ### 3. Creating a new "Phase" for analysis Suppose you create a new phase with the last 7 days of an experiment. Doing this will mean that GrowthBook cannot deduplicate users across the full history of the experiment and the users that show up in the last 7 days of an experiment won't be randomly assigned across variations. If any variation causes more return users, then you'll get carryover bias. In this case, rather than utilize a "Phase" to analyze a particular date range in an experiment, you should instead focus on your **Metric** date windows (not **Experiment** date windows, which are controlled by Phases). There will be more work in the future to make this easier to achieve within GrowthBook, but for now you can: - Add a "metric delay". You can use the **metric delay** setting in GrowthBook to delay tracking metric data until a number of hours or days after an exposure to prevent novelty effects. - Use a "lookback window". In your metric settings, choose a "lookback window" and set it to 7 days. --- ## Holdout Experiments ## What are holdout experiments? Holdout experiments (or simply "holdouts") measure the long-term impact of features by maintaining a control group that doesn't receive new functionality. While most users experience your latest features and improvements, a small percentage remains on the original version, providing a baseline to measure cumulative effects over time. ## Who uses holdouts? Major tech companies often use holdouts to measure both individual feature impact and overall product evolution. However, we recommend that teams of all sizes use holdouts at some level, even if only occasionally. Starting with a simple, single-feature holdout helps you understand the feature's long-term impact and reveals how experiment effects persist at your company. ## Why run a holdout? - **Measure long-term impact**: The impact of features changes over time, as does user behavior, and running a holdout experiment for an extended period will help you understand these effects. - **Understand cumulative effects**: Features interact in unexpected ways, and these interactions shift over time. Holdouts capture the combined impact of multiple changes, revealing synergies and conflicts that individual experiments miss. Some approaches to measuring cumulative effects involve summing up the effect of individual experiments. Often, analysts then shrink this total by some amount to account for statistical bias (like we do on our (impact dashboard)[/insights#scaled-impact]) and to account for the general understanding that long term impacts often shrink or cannibalize one another. But even then, often these adjusted totals still overstate or mis-state the total impact (for example, see [this analysis by AirBnB](https://medium.com/airbnb-engineering/selection-bias-in-online-experimentation-c3d67795cceb)). We could then add, if we wanted, or we could cut this altogether and save this for somewhere else: Holdouts expose users to a combined set of features over a long period of time, naturally measuring their interactions (positive and negative) and serving as a back-test that deals with the selection bias issue in summing up shipped experiments. For this reason, they are the gold standard for measuring cumulative, long-run impact. ## Why wouldn't I want to run a holdout? - **User experience trade-offs**: Holdouts require some users to miss out on improvements and new features while serving as the control group. - **Technical overhead**: Holdouts require you to maintain feature flags in your codebase for their duration. - **Authentication considerations**: Holdouts work best for logged-in experimentation, but can still be useful with anonymous traffic. ## Can I run a holdout experiment in GrowthBook? **Yes!** See our docs on [how to run a holdout experiment](/app/holdouts) in GrowthBook. --- ## Troubleshooting Experiments ## Problem 1: No or little traffic flowing into the experiment There are a few common reasons why there may not be any data showing in the Results tab of your experiment's detail page, first among them is a lack of experiment traffic. You'll likely see a banner like this: ![No Data Banner](/images/no-data-banner.png) Check the following to correct a lack of traffic to the experiment: - **Ensure the trackingCallback in the SDK is configured correctly.** It should be using the intended tracking library (e.g. Google Analytics [GA4], Segment, Amplitude) and actually emitting tracking events to your data warehouse. - **Ensure the Hash Attribute identifier is being set for each user.** When you start an experiment in GrowthBook, you need to choose which Attribute to use to assign users to variations, often the user `id`. If you do not set that Attribute when you instantiate the GrowthBook SDK, then users can't be assigned to the right variation and users will default to not entering the experiment at all. - **Remove any Segments or Activation Metrics that may be filtering out users.** Open the Analysis Settings for your experiment (click the **Edit Settings** button — only visible if you have edit permissions, see [User Permissions](/account/user-permissions)) and remove any selected Activation Metric or Segment. These may be filter out users. If removing them causes data to show up, you need to investigate your configuration of that Activation Metric or your Segment. - **Ensure the Experiment Assignment Query in your data source is configured correctly.** You can find this query on your data source's detail page. ## Problem 2: SRM errors (traffic imbalance) in the experiment A clue that the traffic to the experiment isn't balanced is the presence of a Sample Ratio Mismatch error, shown as a warning banner at the top of the Results tab. ![SRM Warning](/images/srm.png) Whenever there is traffic imbalance in your GrowthBook experiment, the cause is usually one (or several) of these problems: 1. **Decoupled assignment and tracking**: The experiment assignment (e.g., server-side) and exposure tracking event (e.g., client-side) are happening in different places. 2. **Misconfigured `trackingCallback`**: The `trackingCallback` is configured to send an event to your data warehouse _only under certain conditions_. 3. **Misconfigured Activation Metric**: You are using an Activation Metric that is triggered differently by users across variations.. 4. **Hash attribute mismatch**: The hash attribute doesn't match the "Identifier Type" in your data source. 5. **Mid-experiment Targeting Changes**: You made a change to the experiment targeting after it started, or created a new phase without re-randomizing. ### Solution 1: Couple the tracking event with assignment _This solution addresses Problem 1: Decoupled Assignment and Tracking._ This is the most common and severe cause of SRM. While it's tempting to fire the exposure event on the frontend (e.g., "when the component is viewed") to be sure the user saw the change, this approach makes your experiment data vulnerable to a number of client-side issues: - **Bot & Crawler Traffic**: Bots will hit your server, be assigned a variation, but will never execute your client-side JavaScript code. This means that they are counted on the backend but will never fire the frontend event. Depending on your site, this can be 50% or more of traffic. - **Ad Blockers & Privacy Tools**: A large percentage of real human users (often 30-40%) use blockers that stop analytics and tracking scripts (like Segment, Google Analytics, and others) from executing. These users are also assigned but will never fire the frontend event. - **Client-Side Factors**: Normal users may bounce before the page loads, have a network error, or encounter an unrelated JavaScript error that prevents your tracking script from firing. This massive drop-off becomes an SRM error when it is differential—that is, one variation causes more drop-off than another (e.g., the variant is 100ms slower, causing more users to bounce before the event fires). The most robust solution is to fire your exposure tracking event from your backend, immediately after you get the variation assignment from the SDK. This couples assignment and tracking into a single, reliable step. It completely bypasses the client-side issues, guaranteeing that every user who is assigned a variation is also counted in the experiment. A common concern with this approach it that it "dilutes" metrics by including users who were assigned but (due to bouncing, etc.) never saw the change. This is often a feature, not a bug! If your variant causes 2% of users to bounce because it's slow or buggy, that is a critically important negative impact of your experiment. Backend tracking correctly captures this negative impact. Frontend-only tracking hides this problem and biases your results. If you absolutely must filter for users who saw the change, use this pattern: 1. Fire the main exposure event from the backend. This will be your reliable denominator, and it will not have SRM. 2. Fire a separate frontend event (e.g., component_viewed) when the user actually sees the component. 3. In GrowthBook, use this frontend event as an Activation Metric. This way, your experiment's health (the SRM check) is validated on the reliable backend data, but your final metric results are filtered to only the "activated" users you care about. ### Solution 2: Ensure the trackingCallback is triggered unconditionally _This solution addresses Problem 2: Biased trackingCallback Logic._ The `trackingCallback` you configure is used by the SDK to send tracking events to your data warehouse whenever a user is exposed to an experiment. Make sure that whether the `trackingCallback` method sends event data is _not conditional_ on the experiment variation. As you create the GrowthBook-related logic in your application code, be careful not to have users in control variations emit trackingCallbacks under more conditions than users who are _not_ bucketed into the control variation. Otherwise, you are likely to have Sample Ratio Mismatch (SRM) issues. It is much better to emit a tracking event as soon as possible within the trackingCallback, before doing any other custom logic that may differ across users based on the experiment variations they are exposed to. ### Solution 3: Only use Activation Metrics that do not differ across variations _This solution addresses Problem 3: Biased Activation Metric._ If you are using an Activation Metric to filter users that get into your experiment analysis, you may be introducing bias if the Activation Metric is downstream of any differences between users in your variations. Similar to the above issue where your trackingCallback fires more for users in one variation than another, if you filter out users based on some activity that differs across variations then you could be introducing bias and causing SRM errors. The upshot is that you should not use an Activation Metric lightly. It should be a Metric that is downstream of changes to the user experience caused by the experiment. Imagine you want to check a user variation on page load, but you want to use an Activation Metric to only look at effects once users open the checkout modal where the experiment is. If you have to load more data for users in your test variation, the page may crash or be slower for those users; if this happens before you measure the Activation Metric, then you may be causing traffic imbalance as you will see fewer users in the variation in your experiment sample. In fact, using an Activation Metric that is downstream of any difference in variations can cause bias that is not picked up by SRM errors, so you should try to avoid this whenever possible. Best practices: - Try not to use Activation Metrics at all, and instead only use the SDK to look up experiment membership as close as possible to the actual experiment. - If you must use an Activation Metric, make sure it is not downstream of any differences in the user experience caused by the experiment. - Check the Health Tab on your experiment results to see traffic breakdowns by Activation Status to help you understand what your Activation Metric is doing to your sample. ### Solution 4: Ensure the Hash Attribute and Identifier Type match _This solution addresses Problem 4: Hash Attribute Mismatch._ GrowthBook randomizes users according to the Hash Attribute that you set for each user, often a `user_id`. When you analyze experiments, you set an "Identifier Type" that corresponds to columns in your data warehouse. Ideally, the Hash Attribute that you choose for users is both (1) logged to the data warehouse in the trackingCallback, and (2) is used as the "Identifier Type" for all of these experiments. By doing this, you establish a 1:1 relationship between the Hash Attribute and the Identifier Type. For example, suppose you run experiments and you sometimes randomize users by `device_id` and sometimes randomize by `user_id`, setting the Hash Attribute as `device_id` or `user_id` respectively. You would set up your trackingCallback so that the Identifier Types are logged to your data warehouse with each experiment exposure. Then, you would also have two Identifier Types in your data source, one for each Hash Attribute (`device_id` and `user_id`), and define Experiment Assignment Queries to query your data source for all the experiments of each Identifier Type. If for some reason the Identifier in your data warehouse is different from your Hash Attribute, you could see Multiple Exposure warnings and Sample Ratio Mismatch (SRM) errors, because the ID chosen to randomize users doesn't match the ID on which queries are running in your data warehouse. You would fix this so that there is a 1:1 mapping between the Identifier and Hash Attribute by ensuring the Hash Attributes are always emitted to the data warehouse, and use set these values as the Identifier Types in GrowthBook. **Why might the Hash Attribute and Identifier Type be different?** There are times where it might be beneficial to have the Hash Attribute differ from the Identifier Type. The consequence is that there may be minor traffic imbalance or Multiple Exposures, but you may be willing to accept some slippage in the 1:1 mapping between Identifier Type and Hash Attribute in order to more easily join experiment exposure data with metric data. For example, on client-side experiments with Google Analytics (GA4), you may wish to avoid flickering by hashing on some custom ID as we do in our [Script Tag SDK](/lib/script-tag) and not wait for the IDs provided by GA4. While you could send this custom ID in your trackingCallback you may prefer to use GA4 identifiers as the Identifier Type instead, because they are already tracked with many of the other GA4 events that you are using as Metrics. ### Solution 5: Minimize Experiment targeting changes while running _This solution addresses Problem 5: Mid-experiment Targeting Changes._ When you make changes to the targeting of an experiment or created a new phase without re-randomizing users, it's possible that the traffic to the experiment will become imbalanced. This is because the users who were already in the experiment may continue to be in the same variation at differing rates. Whenever creating a new phase, you should re-randomize traffic when possible, and only certain changes to an experiment's targeting conditions can be done without re-randomizing. You can read more about the [rules for changing experiment targeting](/app/making-experiment-changes) and [carryover bias](/kb/experiments/carryover-bias) in our documentation. If your experiment is suffering from these issues, unfortunately the only solution may be to restart the experiment (you can do this by creating a new phase and choosing to re-randomize traffic). Of course, some amount of bias and SRM may be preferable to throwing out your data, but this depends on the magnitude of the problem. ### Solution 6 (URL Redirects): Ensure trackingCallback fires If you're seeing SRM warnings while running a URL Redirect test, especially if you are using a front-end SDK such as the HTML, JavaScript, React, or Vue.js SDKs, please review the [Sample Ratio Mismatch (SRM) Warnings on URL Redirect Tests](/app/url-redirects#sample-ratio-mismatch-srm-warnings-on-url-redirect-tests) documentation. ## Problem 3: Multiple Exposures We default to allowing 1 percent of units (e.g. users) in an experiment have multiple exposures without raising a warning due to the fact that many experimenters have slight mismatches in hashing attributes and warehouse tracking that can cause minor errors. You can adjust this in your organization settings under Settings > General > Experiment Settings. If you have multiple exposure errors, the issue is fundamentally caused by a mismatch between the attribute that you are randomizing on (your "hash attribute") and the identifier being stored in your data warehouse. You can [read more about this problem and how to solve it here](#solution-4-ensure-the-hash-attribute-and-identifier-type-match). Another potential issue is that you made unsafe changes to your experiment targeting and randomization. We attempt to prevent this by having a curated flow for making changes, but we provide you with flexibility to make a whole host of changes that could result in multiple exposures if you choose to override our warnings. You can read more about our [rules for changing experiments while running here](/app/making-experiment-changes). ## Problem 4: Metric averages don't look as expected It's important to ensure that the Metric values in the baseline and variation are what you would expect. This helps confirm that the Metrics are set up correctly and that they are joining properly to the Experiment Assignment Queries. In the image below, notice that the values look like reasonable, per-user averages. If there is no data or if the values are very different from what you'd expect, consider the solutions below. ![Reasonable Metric Averages](/images/reasonable-metric-averages.png) ### Solution: Understand the experiment's population Sometimes the Metric values in an experiment may surprise you if you are targeting an experiment to new users (values may be low!) or power users (values may be high!). Each experiment can have different targeting Attributes, and it is totally possible that the values are representing that experiment's population. Consider this before launching into in-depth investigations of your Metric's set-up. ### Solution: Check the Metric's configuration You can confirm that the Metric is returning the right values in the table rows on a per-user basis. Navigate to the Metric page, edit the SQL for the Metric, and use the Edit SQL interface to run test queries and make modifications to make sure the rows you are getting are what you expect. The results should show the right identifiers, timestamps, and values. You should also make sure the [Metric Window](/app/metrics#metric-windows) is not too short and not too long, otherwise it could be excluding or including too much data. ### Solution: Make sure your Identifier Types are correct If you are seeing some users flow into your experiment but aren't seeing any Metric data, you should ensure that the Metric data is able to join to your experiment exposure data. This is illustrated in the image below. Notice the 38,919 total users in the upper left corner, but no data in the "Baseline", "Variation", or "Chance to Win" columns: ![No Metric Values](/images/no-metric-values.png) GrowthBook relies on the Identifier Type in your data source to merge users in your experiment with Metric values. This is accomplished through the Experiment Assignment Query (for users) and the Metric (for Metric values). These Identifier Types must be the same, or linked by a Join Table, to run a query in GrowthBook. While we actually will throw an error if you can't join them, you may still have set up one or the other incorrectly to point to the wrong column, or the data in the columns for your Experiment Assignment Query and Metric may not overlap. It's a good idea to ensure that the values being produced by both queries are what you expect. ### Solution: Understand the overall query logic better You can view the queries generated by GrowthBook by navigating to the experiment's Results tab, looking on the right side of the page (shown below) and clicking "View Queries". You can then use parts of these queries and debug them by running raw SQL in your own data warehouse to make sure that your configuration is correct. ![View Queries Button](/images/view-queries-button.png) --- ## Glossary **A** [A/A Experiment](/kb/experiments/aa-tests): A/A testing is a form of A/B testing where two identical versions of a product or design are tested against each other to check the consistency of the testing platform and methodology. [A/B Experiment](/using/fundamentals#anatomy-of-an-ab-test): A/B testing is a method of comparing two versions of a webpage or other product to determine which one performs better by splitting traffic between the two versions and measuring their performance using statistical analysis. [Analysis attribute](/lib/python#attributes): Analysis attribute in GrowthBook is a user-specific attribute (like user id, device id, etc.) used for segmenting and analyzing experiment results based on different user characteristics. [Approval Flow](/features/publishing-and-approval-flows): Approval flow is a feature in GrowthBook that requires changes to features to be reviewed and approved before they are published, reducing errors and ensuring changes have been viewed by someone else in the organization. [Assignment](/experiments#custom-assignment-or-3rd-party-experiments): Assignment in GrowthBook refers to the process of allocating users to different variations of an experiment, typically done using a consistent hashing algorithm to ensure the same user always gets the same variation given the same experiment settings. [Assignment attribute](/app/experiment-configuration#experiment-targeting-and-traffic): Assignment attribute in GrowthBook is a user-specific attribute (like user id, device id, etc.) used to consistently assign the same user to the same variation across multiple sessions or devices. [Audit Logs](/account/audit-logs): Audit logs in GrowthBook are records of all actions taken on the platform, useful for auditing user activities and troubleshooting issues. These logs can be accessed from the Settings → Log in the left navigation. [Audit Logs Export](/account/audit-logs): Audit logs with the ability to search and export. **B** [Binomial Metric](/app/metrics/legacy#binomial-metrics): A binomial metric in GrowthBook is a type of legacy metric that represents a binary outcome (yes/no or success/failure), such as whether a user converted or not, and is used to compute the proportion of users in an experiment variation who converted. The new name for this metric type is a Proportion Metric. **C** [Carryover Bias](/kb/experiments/carryover-bias): Carryover bias in GrowthBook refers to the contamination of experiment results when users are not re-randomized between different phases of an experiment, causing the effects of the first phase to influence the results of the subsequent phase. [Cloud-Hosted](/quick-start#set-up-growthbook): Cloud-hosted in GrowthBook refers to the GrowthBook platform hosted on the cloud, providing a managed service that takes care of infrastructure, maintenance, and updates, allowing users to focus on running and analyzing experiments. [Code Reference](/features/code-references): Code Reference in GrowthBook allows teams to quickly see instances of feature flags being leveraged in their codebase, helping surface valuable information such as flagging stale feature flags more accurately, and directing devs to the exact lines of code that need to be cleaned up. [Config.yml File](/self-host/config): The `config.yml` file in GrowthBook is used to define data sources, metrics, and dimensions, making it easier to version control the configuration and share between environments when self-hosting the platform. [Control](/using/fundamentals#control-or-baseline): The control in GrowthBook refers to the baseline or original version in an A/B test against which other variations are compared to determine their effectiveness. [Count Metric](/app/metrics/legacy#conversion-types): A count metric in GrowthBook is a type of legacy metric that represents the number or magnitude of conversions (like pages per visit or points earned), where the value of each conversion event is summed together at the user level before taking an average per variation. The new name for this metric type is a Mean Metric. [CUPED](/statistics/cuped): Enable CUPED to reduce variance in experiments and get results up to 2x faster. [Custom CDN Usage Limits](https://www.growthbook.io/pricing): Pro and Enterprise organizations can pay for additional CDN usage above the included amount (varies by plan, starts at 1M requests / 5GB bandwidth) on GrowthBook Cloud. [Custom Fields](/using/growthbook-best-practices#custom-fields): Allows enterprise organizations to create custom fields that can be added to features and experiments. [Custom Page Markdown](/using/growthbook-best-practices#custom-markdown): Inject custom content onto specific pages to help enforce company standards and processes. For example, add a link to internal best practices at the top of all feature flag pages. [Custom Pre-Launch Experiment Checklist](/app/pre-launch-checklist): Create custom checklists that have to be completed before starting an experiment. For example, require users to add a hypothesis. [Custom Roles](/account/user-permissions#custom-roles): Define your own user roles with custom sets of permissions **D** [Data Pipeline Mode](/app/data-pipeline): Write intermediate experiment results to temp tables in your data warehouse for more efficient and faster data refreshes [Data Source](/app/datasources): A Data Source in GrowthBook is how the platform connects to your data warehouse to pull aggregated statistics in order to compute metrics and experiment results, supporting various SQL versions and event tracker software. [Data Warehouse](/warehouses): A Data Warehouse in GrowthBook is your secure storage where your data resides, and only aggregated statistics are transmitted to GrowthBook servers or your self-hosted environment for analysis. [Dimension](/app/dimensions): A Dimension in GrowthBook is a way to slice and dice your data, allowing you to drill down into your results by different user attributes or experiment characteristics for deeper analysis. [Discover Metrics / Auto Generate Metrics](/app/metrics/legacy#auto-generate-metrics): Allows organizations using GA4, Segment, Rudderstack, or Amplitude for event tracking to automatically generate count and binomial metrics for each unique tracked event. [Duration Metric](/using/experimenting#test-duration): A duration metric in GrowthBook is a type of metric that measures the time it takes to do something (like page load time or time on site), where the value of each event is summed together at the user level before taking an average per variation. **E** [Edge SDK](/lib/edge/cloudflare): The Edge SDK in GrowthBook is a set of libraries designed to run on edge workers, enabling server-side or hybrid experimentation with features like visual and URL redirect experiments without flicker or delay. [Encrypted SDK Payload](/app/api#encryption): Provide an extra level of obfuscation to avoid leaking sensitive feature flag and experiment configs in client-side integrations. [Enterprise Support](https://www.growthbook.io/pricing): Shared channel in Slack or Teams with a dedicated Account Manager and Support SLAs. [Environment](/features/environments): An Environment in GrowthBook is a specific context (like production, staging, or development) where you can enable or disable feature flags, allowing you to control and segment feature delivery across different stages of your application lifecycle. [Environment Inheritance](/features/environments#environment-inheritance): Create environments based on a parent environment to inherit all of its feature rules at the time of creation. Can be used for setting up new persistent environments (e.g. staging) or temporary ones for troubleshooting without affecting the parent. [Environment-Scoped Permissions](/account/user-permissions#environment-specific-limits): Allow a user to publish feature flag changes only to selected environments (e.g. dev, but not production). [Event Webhooks](/app/webhooks/event-webhooks): Event Webhooks in GrowthBook are outbound notifications that trigger when the state of GrowthBook changes, such as when a feature is modified or a new experiment is created, enabling real-time communication from GrowthBook to your applications. [Experiment Assignment Query](/app/datasources#experiment-assignment-queries): An Experiment Assignment Query in GrowthBook is a SQL query that returns which users were part of which experiment, what variation they saw, and when they saw it, helping to generate experiment results. [Experimentation](/using/experimenting): Experimentation in GrowthBook refers to the process of testing different variations of a feature or product to determine which one performs better, using statistical analysis to make data-driven decisions. **F** [Fact Table](/app/metrics): A Fact Table in GrowthBook is a feature that allows you to write SQL once for an event and from that, you can create multiple related metrics, simplifying the process of defining metrics and enabling advanced performance and cost optimizations. [Fact Table Optimization](/kb/metrics/fact-metric-optimization): Combine multiple metrics from the same Fact Table into a single query, drastically cutting down on data warehouse compute and execution time. [Feature](/features/basics): Features in GrowthBook are configurable elements of your product that can be controlled via the GrowthBook UI, allowing you to change your application's behavior, run experiments, and gradually roll out new functionality to your users. [Feature Flag](/features/basics): A Feature Flag in GrowthBook is a powerful developer tool that allows you to control how and when new functionality is released to your users, supporting different types (Boolean, Number, String, JSON) and environments (dev, production). [Feature Key](/features/basics#feature-keys): A Feature Key in GrowthBook is a unique identifier that you reference in your application to control a specific feature flag, and it cannot be changed later once it's created. [Forced Value](/features/rules#forced-value): A Forced Value in GrowthBook is a type of feature rule that assigns a specific value to a feature for a subset of users based on targeting attributes, useful for testing or targeting specific user groups. **G** [Global SDK Webhooks](/app/webhooks/global-sdk-webhooks): Global SDK Webhooks in GrowthBook are notifications that fire for all SDK Connection changes, not limited to a single SDK Connection, and are particularly useful for larger organizations with multi-org installations of GrowthBook. [GrowthBook DevTools Browser Extension](/tools/chrome-extension): The GrowthBook DevTools Browser Extension (Chrome and Firefox) integrates with the JavaScript SDKs, allowing you to QA and debug feature flags and experiments from GrowthBook's JavaScript SDKs directly in your browser. [GrowthBook Proxy](/self-host/proxy): The GrowthBook Proxy is a server that sits between your application and GrowthBook, providing speed, scalability, security, and real-time feature rollouts by caching feature lookups and streaming updates in real-time. **H** [Hashing Algorithm](/lib/js#hashing): The Hashing Algorithm in GrowthBook is a deterministic method used to consistently assign users to experiment variations, ensuring the same user always gets the same variation given the same experiment settings. [Holdout Experiments](/kb/experiments/holdouts): Holdout Experiments in GrowthBook are a method of measuring the long-term impact of one or more features by keeping a set of users (the holdout group) from seeing new features and comparing their behavior to another set of users who do see the new features. **J** [JSON Schema Validation](/features/basics#json-flags): Apply validation to JSON feature flags to enforce a specific data structure and avoid typos. **L** [Live Chat Support](https://www.growthbook.io/pricing): An in-app messaging tool for providing instant customer support and assistance. GrowthBook Cloud only. **M** [Mean Metric](/app/metrics#mean-metrics): A Mean Metric in GrowthBook is a type of metric that measures an average value among all experiment users. For example, the average revenue per user. [Metric](/app/metrics): A Metric in GrowthBook is a key performance indicator that your experiments are trying to improve, defined via SQL or a simple query builder, and can be of various types like proportion, mean, quantile, or ratio. [Metric Groups](/app/metrics#metric-groups): Group related metrics together and easily add them in bulk to experiments. Great for company-wide guardrail metrics. [Metric Overrides](/app/metrics#bayesian-priors): [Metric Query](/app/metrics/legacy#query-settings): A Metric Query in GrowthBook is a SQL or simple query that defines how to fetch data for a specific metric from your data source, returning one row per "conversion event" and supporting various types of identifiers. [Multi-Arm Bandits](/bandits/overview): Optimize experimentation by dynamically adjusting traffic to the best-performing variations in real-time. [Multi-Org Deployments](/self-host/env#multi-org): For self-hosted deployments, configure multiple top-level organizations with complete data isolation. Users can be centrally managed and assigned to multiple orgs. [Multiple Exposures](/using/experimenting#multiple-exposures): Multiple Exposures in GrowthBook refers to a situation where a substantial number of users in an experiment have been exposed to multiple variations, indicating a potential issue with the experiment assignment or tracking implementation. [Multiple SDK Webhooks](/app/webhooks/global-sdk-webhooks): The ability to fire multiple webhooks when an SDK payload changes. This enables complex integrations and workflows. **N** [Namespace](/features/feature-flag-experiments#namespaces): A Namespace in GrowthBook is a feature that allows you to make multiple experiments mutually exclusive, ensuring that users are only part of one experiment within the same namespace at a time, useful for avoiding conflicts between experiments. [No Access Role](/account/user-permissions#how-does-the-no-access-role-work): A base role with zero permissions (not even read access). Combine with project-scoped roles for fine-grained access control. **O** [Onboarding and Training](https://www.growthbook.io/pricing): Help your team be successful with onboarding presentations and team training customized to your needs. Our team of experimentation experts can help elevate your team to understand A/B testing best practices and pitfalls, and run better experiments. For Enterprise plan customers. [Overall Experiment Impact](/insights#dashboard): Quantify ROI of your experimentation program by rolling up the impact of all experiments within a time period. **P** [Peeking Problem](/statistics/sequential#the-peeking-problem): The Peeking Problem in GrowthBook refers to the issue of experimenters making decisions about the results of an experiment based on early data, which can lead to inflated false positive rates, and can be mitigated using techniques like Sequential Testing. [Percentage Rollout](/features/rules#percentage-rollout): A Percentage Rollout in GrowthBook is a rule that allows you to gradually release a feature to a random sample of your users, useful for ensuring a new feature doesn't break your app or site and for controlling the exposure of your users to the new feature. [Prerequisite Feature](/features/prerequisites): A Prerequisite Feature in GrowthBook is a feature that controls the state of other features, rules, and experiments based on its own state, allowing for grouping of related features, creating a hierarchy of features, and enabling features based on user bucketing. [Prerequisite Targeting](/features/prerequisites#inline-prerequisite-targeting): Target individual experiments or feature flag rules based on the value of a different feature flag. [Project](/using/growthbook-best-practices#projects): A Project in GrowthBook is a way to organizationally separate features, metrics, experiments, and data sources by team or product feature within an organization, allowing for focused views and management of specific sections of GrowthBook. [Project-Scoped Roles](/account/user-permissions#project-specific-permissions): Assign a user different roles based on project. For example, read-only access globally, but full write access to specific projects. [Proportion Metric](/app/metrics#proportion-metrics): A Proportion Metric in GrowthBook is a type of metric that measures the percent of experiment users who match a specific criteria (like the percent of users who purchased something). **Q** [Quantile Metric](/app/metrics#quantile-metrics): A Quantile Metric in GrowthBook is a type of metric that measures a specific quantile (Median, P90, etc.) of a value. For example, P99 Latency or Median Revenue per User. **R** [Ratio Metric](/app/metrics#ratio-metrics): A Ratio Metric in GrowthBook is a type of metric that measures the ratio between two values. For example, Average Order Value (sum of revenue divided by number of orders). [Remote Evaluation](/lib/js#remote-evaluation): Allows experiments and feature flags to be evaluated on the server-side, ensuring consistent results across all client environments. This approach improves security and reduces the risk of manipulation or misconfiguration on the client-side. [Revenue Metric](/app/metrics/legacy#conversion-types): A Revenue Metric in GrowthBook is a type of legacy metric that measures the amount of revenue earned (like revenue per user or average order value), where the value of each event is summed together at the user level before taking an average per variation. The new name for this is a Mean Metric. [Roadmap Acceleration/Custom Features](https://www.growthbook.io/pricing): Get custom features built or ask us to accelerate feature development from our roadmap. For Enterprise plan customers. [Rule](/features/basics#rules): A Feature Rule in GrowthBook is a tool to change the default value of a feature for specific users or groups, supporting different types like Forced Value, Percentage Rollout, and A/B Experiment, and providing fine-grained control over feature assignment. **S** [Sample Ratio Mismatch (SRM)](/using/experimenting#sample-ratio-mismatch-srm): A Sample Ratio Mismatch (SRM) in GrowthBook is a health check that warns you when the actual traffic split between experiment variations significantly deviates from the expected split, indicating potential issues with the experiment configuration or implementation. [Saved Groups](/features/targeting#saved-groups): Saved Groups in GrowthBook are reusable lists of users, based on specific attributes, that can be targeted across multiple features or experiments, supporting two types: ID Lists and Condition Groups, and allowing for efficient and consistent user targeting. [Saved Group Performance](/features/targeting#id-lists): Improved SDK performance when using saved groups to target large lists of user ids. [Scheduled Changes](/features/rules#scheduling-rules): Launch or end experiments and feature flags at pre-set times. [Schema Browser](/app/datasources#schema-browser): Allows a user to view the databases, schemas, and tables of their connected data source without having to leave GrowthBook. It also outlines what rows are available on a particular table, and the data type of each row. [SCIM](/integrations/scim): SCIM (System for Cross-domain Identity Management) in GrowthBook is a standard for managing users and groups across multiple applications, enabling automation of user provisioning and deprovisioning through your identity provider, currently supporting Okta as the identity provider. [SDK](/lib/): The SDK (Software Development Kit) in GrowthBook is a set of libraries for different programming languages that allows you to integrate GrowthBook into your application, enabling feature flagging, running experiments, and sending data back to GrowthBook for analysis. [SDK Connection](/features/environments#environments-and-sdks): An SDK Connection in GrowthBook is a unique link between your application and GrowthBook, generating a clientKey for read-only access to feature flags in a specific environment, and used by the SDKs to fetch feature flag states and rules from the GrowthBook API. [SDK Connection Endpoint](/app/api#sdk-connection-endpoints): The SDK Connection Endpoint in GrowthBook is a readonly API endpoint that provides access to a subset of your feature flag data, just enough for the GrowthBook SDKs to assign values to users, and does not require authentication to view. [SDK Webhooks](/app/webhooks/sdk-webhooks): SDK Webhooks in GrowthBook are notifications that trigger a script on your server whenever something changes within GrowthBook that will affect that SDK, allowing for real-time updates and synchronization of feature flag states. [Secure Attributes](/lib/js#secure-attributes): Mark specific targeting attributes (e.g. email address) as "secure" and they will be hashed before being sent to client-side SDKs [Segment](/using/experimenting#digging-deeper) (deprecated): A Segment in GrowthBook was a subset of users that match a particular attribute, allowing you to filter experiment results to only show users that match the segment, created with SQL from the "Data and Metrics → Segments" page. Segments have been deprecated and are being phased out; if you don't already have one, use a [Dimension](/app/dimensions) or a custom SQL filter in the experiment's analysis settings instead. [Self-Hosted](/self-host): Self-Hosted in GrowthBook refers to the option of hosting the GrowthBook platform on your own infrastructure, giving you full control over your data and allowing you to customize the platform to suit your specific needs. [Sequential Testing](/statistics/sequential): Allows for continuous monitoring and analysis of experiments, allowing early decisions without compromising statistical validity. [Simulate Feature Flags](/features/rules#testing-rules):For any set of user attributes, see how feature flags would be evaluated for that user. Save common sets of attributes as "archetypes" for easier debugging. [SSO](/sso): Single Sign-On (SSO) in GrowthBook is an enterprise feature that allows users to authenticate with a single set of credentials across multiple applications, available on GrowthBook Cloud or Self-hosted via OpenID Connect, and configurable with various identity providers. [Stale Feature Flag](/features/stale-detection): A Stale Feature Flag in GrowthBook is a feature flag that is no longer actively used or relevant, and GrowthBook's Stale Feature Flag Detection feature helps identify and manage these flags, ensuring your feature flag ecosystem remains clean and efficient. [Sticky Bucketing](/app/sticky-bucketing): Sticky Bucketing in GrowthBook is a feature that ensures users continue to see the same variation when changes are made to a running experiment, allowing for consistent user experiences and accurate experiment results. **T** [Targeting Condition](/features/targeting#targeting-conditions): A Targeting Condition in GrowthBook is a rule that determines which users are included in an experiment or receive a specific feature value, based on user attributes, and can be defined using a simple UI or advanced JSON syntax inspired by MongoDB query syntax. [Team Permissions](/account/user-permissions#teams): Create teams of users and manage permissions and access on a team level [Top-Level Prerequisites](/features/prerequisites#top-level-prerequisites): Define a parent/child relationship between feature flags. When the parent is toggled off, all children flags will also be off. [TrackingCallback()](/lib/build-your-own#trackingcallback): The `trackingCallback()` in GrowthBook is a function that is called every time an experiment is run, allowing you to record the assigned value in your event tracking or analytics system, and is used across various SDKs for tracking experiment results. [Traffic Split](/lib/node#customizing-the-traffic-split): A Traffic Split in GrowthBook is a configuration that determines how users are divided among different variations in an experiment, allowing you to control both the percentage of users included in the experiment and the distribution of users among the variations. **U** [Uptime SLA (99.99%)](https://www.growthbook.io/pricing): For cloud customers, we offer uptime guarantees at four 9s. [URL Redirect](/app/url-redirects): A URL Redirect in GrowthBook is a type of experiment that redirects users to different URLs based on their assigned variation, ideal for testing big changes or complete page redesigns, and requires integration with GrowthBook's SDKs. [URL Split Test](/using/experimentation-problems#redirect-tests-split-testing): A URL Split Test in GrowthBook is a method of A/B testing where traffic is split between two different URLs to compare their performance, typically implemented using feature flags and custom JavaScript to redirect users based on their assigned variation. **V** [Variation](/using/fundamentals#variations): A Variation in GrowthBook is a version of a feature or product that is being tested in an experiment, with users being assigned to different variations to compare their performance and impact on key metrics. [Visual Editor](/app/visual): The GrowthBook Visual Editor is a Chrome side-panel extension for designing A/B tests without writing code. Install it from the [Chrome Web Store](https://chromewebstore.google.com/detail/growthbook-visual-editor/nbomejknbpkcpjdagefhichaajpoempk). --- ## Google Analytics (GA4) Troubleshooting # GrowthBook and Google Analytics (GA4) Google Analytics (GA4) is a popular Data Source among GrowthBook users. This article helps troubleshoot common issues encountered when integrating GA4 with GrowthBook, and can be used in combination with our guide on [A/B Testing with Google Analytics 4 (GA4) and GrowthBook](/guide/GA4-google-analytics). Please watch the [How GrowthBook Works with Your Data](https://youtu.be/hboLeUbpdus) (4 minutes) overview video to better understand the issues and solutions in this troubleshooting guide. If you are using Google Tag Manager (GTM) in combination with GA4, please review our [Google Tag Manager Setup Guide](/guide/google-tag-manager-and-growthbook) and accompanying [Google Tag Manager (GTM) Troubleshooting Guide](/kb/google-analytics/google-tag-manager-gtm-troubleshooting). ## Special note for the HTML and JavaScript SDKs ### GA4 and GrowthBook's HTML Script Tag SDK GrowthBook's HTML Script Tag SDK provides [automatic experiment tracking](/lib/script-tag#tracking-experiment-views) if it detects Google Tag Manager (GTM) and/or Google Analytics (GA4) on the webpage. For each experiment view, the HTML SDK sends an event with the `experiment_id` and any `variation_id`s to these analytics tools if they are present on the webpage. No additional configuration is needed for these integrations; simply include the HTML script tag as shown in the documentation and the `experiment_viewed` events will be sent automatically. If you need to define a custom `trackingCallback` you can do so with the `window.growthbook_config.trackingCallback` object. This must be set before loading the script tag. ### GA4 and GrowthBook's JavaScript SDK Similar to the details above regarding the HTML SDK, the JavaScript SDK can accommodate [an optional plugin](/lib/js#third-party-tracking) that will automatically send `experiment_viewed` events to GTM and/or GA4 if configured. The plugin is **not** enabled by default and must be imported and then added to the `plugins` array when constructing the `growthbook` instance. ## Common issues when integrating GA4 with GrowthBook The issues described below are commonly reported and can lead to missing or incomplete experiment data in GA4. ### Data propagation delays There is a known 24-48 hour delay between when events are sent to GA4 and when they appear in your BigQuery instance, which can cause temporary gaps in reporting. Read more about this initial sync delay in our [Google BigQuery (BQ) Troubleshooting Guide](/kb/google-analytics/google-big-query-bq-troubleshooting#initial-sync-delay-for-new-ga4-properties). ### User counts and data discrepancies GrowthBook and GA4 may count users differently due to factors like bot filtering, data processing differences, or sampling in GA4. This can lead to mismatched user or event counts between the two platforms. ### Experiment Assignment Query (EAQ) issues #### The default SQL generated by GrowthBook doesn't match the schema in BigQuery The default [Experiment Assignment Queries](/app/datasources#experiment-assignment-queries) in GrowthBook may not match the connected BigQuery dataset or schema, requiring manual adjustment. If these queries are not set up correctly, GrowthBook will not be able to retrieve experiment data from GA4’s BigQuery export. Access the Experiment Assignment Query for each Data Source by going to: 1. The Left navigation menu 2. Metrics and Data 3. Data Sources 4. Select the data source in question 5. Scroll down to the Experiment Assignment Queries section 6. Click the right-facing arrow on the right side of each Experiment Assignment Query to view the SQL for the query Only GrowthBook users with a user role of Analyst, Experimenter, or Admin can modify the Experiment Assignment Queries. Read more about this in our [User & Team Permissions](/account/user-permissions) documentation. #### Testing the experiment's flow of data After implementing the `trackingCallback`, check GA4 **and** the data warehouse (usually BigQuery if GA4 is also being used) to confirm that the `experiment_viewed` events are being recorded as expected. Since the flow of data is from the experiment to GA4 to BigQuery to the results tab in GrowthBook, if no `experiment_viewed` events are present in GA4, they will not be present in BigQuery or in GrowthBook. You can also test the Experiment Assignment Queries for each Data Source to ensure that they return the expected results. Locate the "Test Query" button for each Experiment Assignment Query and click it. If nothing is returned, try commenting out the lines with `experiment_id` and `variation_id` (one at a time) to see if anything gets returned. This can help narrow down whether the `experiment_viewed` event is making it into GA4. While testing the Experiment Assignment Queries, If you see a message that says `No data yet. Make sure your experiment is tracking properly.` it often means the `trackingCallback` is not firing or the data is not reaching GA4. It can also mean that you are within the 24-48 hour known sync delay between GA4 and BigQuery. Please view these resources for more information: - [No or little traffic flowing into the experiment](/kb/experiments/troubleshooting-experiments#problem-1-no-or-little-traffic-flowing-into-the-experiment) - [Initial sync delay for new GA4 properties](/kb/google-analytics/google-big-query-bq-troubleshooting#initial-sync-delay-for-new-ga4-properties) ### The `trackingCallback` doesn't fire or is misconfigured If the GrowthBook SDK’s `trackingCallback` is not implemented correctly, or if it does not send the `experiment_viewed` event to GA4, experiment exposures will not be tracked, resulting in missing or incomplete data in GrowthBook. The HTML Script Tag SDK handles this automatically and no adjustment should be necessary. Usage of other SDKs requires manually configuring the `trackingCallback` to send `experiment_viewed` events to GA4. Read on for more information. #### Format for `trackingCallback` for pure GA4 integrations (no GTM) Use `gtag` in the `trackingCallback` if Google Analytics 4 (GA4) is loaded directly on your page **and** the `window.gtag` function is available on the web page: ```javascript trackingCallback: (experiment, result) => { if (window.gtag) { window.gtag("event", "experiment_viewed", { event_category: "experiment", experiment_id: experiment.key, variation_id: result.variationId // or result.key depending on SDK version }); } ``` #### Format for `trackingCallback` for combined GA4 and GTM integrations Use `dataLayer` in the `trackingCallback` if you are using Google Tag Manager (GTM) to manage your analytics setup, **or** if `gtag` for GA4 is not available on the web page. This approach is recommended when GA4 is loaded via GTM because: - Sometimes `gtag` may not be defined in this type of setup. - If GA4 is not properly loaded on the page, the `window.gtag` function will not exist, and attempts to send events to `gtag` via GrowthBook's `trackingCallback` will fail silently. - When GA4 is loaded through GTM, there can be timing issues because GTM loads asynchronously. This creates a potential race condition where GrowthBook might initialize before GA4 is fully loaded, which can affect experiment tracking. ```javascript // JavaScript trackingCallback: (experiment, result) => { window.dataLayer = window.dataLayer || []; window.dataLayer.push({ event: "experiment_viewed", experiment_id: experiment.key, variation_id: result.variationId // or result.key depending on SDK version }); } ``` See the [Tracking via `dataLayer` and GTM](/guide/google-tag-manager-and-growthbook#4-tracking-via-datalayer-and-gtm) documentation for more information. ### Date range mismatches If the date ranges selected in GA4 and GrowthBook are not identical, reported metrics and user counts will differ. It's important to note that in some cases, even if the date ranges match, it is still possible for there to be a data discrepancy due to some of the other causes outlined in this troubleshooting guide. ### Consent and cookie banner blocking If a cookie consent banner blocks analytics until consent is given, the `experiment_viewed` event may be sent before consent, causing it to be dropped. Delay initializing the GrowthBook SDK and firing the `trackingCallback` (which will send the `experiment_viewed` event) until after the user has granted consent. This ensures that analytics and experiment exposure events are only sent when it is legally and technically permitted. #### Special instructions if using the HTML Script Tag SDK or the Edge SDK Read more about cookies and GDPR compliance in the docs for the [HTML Script Tag SDK](/lib/script-tag#cookie-policy-and-gdpr) and [Edge SDK](/lib/edge/lambda#cookie-policy-and-gdpr). ## Prerequisite Configurations There are a few things that need to be set up properly in GrowthBook in order to troubleshoot GA4 issues, regardless of whether you are using Google Tag Manager (GTM) with GA4 or not. ### Ensure the GA4 Measurement ID is correct The GA4 Measurement ID is a unique identifier for your GA4 property that's used to send data from your website or app to the correct GA4 account. It looks like "G-XXXXXXXXXX" and is required when configuring GA4 tags in Google Tag Manager or when sending events directly to GA4. You can see how the Measurement ID is used in the [GrowthBook and GTM integration](/guide/google-tag-manager-and-growthbook#4-tracking-via-datalayer-and-gtm) documentation, where it is entered as a configuration value for sending experiment events to GA4. ### Ensure the GrowthBook SDK is properly configured Make sure you have created an SDK Connection in the GrowthBook app under **Left navigation menu** > **SDK Configuration** > **SDK Connections**, and that you are using the correct `apiHost` and `clientKey` values in your SDK initialization code. View [the example initialization code](/lib/js#step-1-configure-your-app) (JavaScript) for more information. #### Verify SDK connection using GrowthBook's DevTools Extension The SDK connection can be further validated by using GrowthBook's [DevTools Browser Extension](/tools/chrome-extension) (available for [Chrome](https://chromewebstore.google.com/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/)), which allows you to inspect experiments and confirm that DOM mutations are being applied as expected. If your site is public, the extension should detect the SDK and display active experiments. If you encounter issues, double-check that your SDK code is correctly implemented and that the site is accessible (not behind authentication or firewalls that could block the SDK). For advanced setups, such as using a proxy or remote evaluation, ensure that the relevant options are enabled both in your SDK Connection and in your SDK initialization (include `remoteEval: true` in the GrowthBook initialization code). ### Ensure the SDK is properly initialized in the application To properly initialize the GrowthBook SDK in your application, you first need to install the appropriate SDK package for your language or framework (e.g., `npm install @growthbook/growthbook-react` for React, or the relevant package for Java, PHP, etc.). Follow the [SDK-specific docs](/lib/) and ensure the initialization code being used contains the correct `api_host`, `client_key`, and that any relevant user attributes are present. #### If GrowthBook is initialized on a specific page (not globally) To initialize the GrowthBook SDK on a specific page, instead of globally, you can create and configure a GrowthBook instance directly within that page’s component or script, rather than at the global application level. For example, in a React or Next.js app, ensure that the GrowthBook SDK has been imported and instantiated inside a page component or within a `useEffect` hook. Of course, make sure that the experiment in question is located on the page that is initializing GrowthBook. ### Ensure the `trackingCallback` is called unconditionally Try not to bury the `trackingCallback` within conditional application logic. Conditionally calling the `trackingCallback` can cause fewer users to be assigned to the experiment than you are expecting. Refer to [this section](/kb/experiments/troubleshooting-experiments#solution-1-ensure-the-trackingcallback-is-triggered-unconditionally) in our Knowledge Base for more information. #### Experiment Rule Precedence Rules associated with a feature in GrowthBook are evaluated from top to bottom as they appear in the GrowthBook UI. When a user matches a rule higher up in the list, they will not be evaluated for any subsequent rules. To ensure that the `trackingCallback` fires for users in an experiment, make sure that the Experiment Rule is the first rule that those users match. Read more about [Experiment Rules](/features/rules#experiments) in our documentation. ### Ensure GA4 is properly configured as the Data Source in GrowthBook Double-check that our [GA4 Setup Guide](/guide/GA4-google-analytics) was followed exactly. Make sure the BigQuery service account used by GrowthBook has the necessary permissions on your BigQuery project (at least BigQuery Data Viewer, Metadata Viewer, and Job User). This allows GrowthBook to read the data in GA4. Test the Experiment Assignment Query as outlined in the [Experiment Assignment Query (EAQ) issues](#experiment-assignment-query-eaq-issues) section above. ### Ensure the experiment is properly scoped #### Environments The environment(s) allowed for the experiment must match the environment(s) allowed for the SDK. #### Projects The project(s) allowed for the experiment must match the project(s) allowed for the SDK. The project(s) allowed for the experiment must also match the project(s) allowed for the Data Source. ## GA4 troubleshooting if **not** using Google Tag Manager (GTM) If you **are** using GTM with GA4 and GrowthBook, please review our [Google Tag Manager Setup Guide](/guide/google-tag-manager-and-growthbook) and accompanying [Google Tag Manager (GTM) Troubleshooting Guide](/kb/google-analytics/google-tag-manager-gtm-troubleshooting) for this section instead. ### Why are there no `experiment_viewed` events showing in GA4? #### Possible Answer #1: GA4 is not loaded or `window.gtag` is unavailable If GA4 is not properly loaded on the page the `window.gtag` function will not exist, and any attempt to send events to it will silently fail. You can check this by opening your browser's developer console and typing `window.gtag`. If the response is `undefined`, you'll need to make some changes to where or how GA4 is loaded in your application to ensure it's available before GrowthBook tries to send events to it. #### Possible Answer #2: No users are being bucketed into the experiment If the GrowthBook SDK is not assigning users to variations (for example, due to missing or incorrect user attributes), the `trackingCallback` will not fire, and no events will be sent. ### Why are there fewer `experiment_viewed` events showing in GA4 than I expect? #### Possible Answer #1: Experiment assignment issues If users are not being assigned to the experiment due to targeting rules, missing attributes, or other logic the `trackingCallback` will not fire, and no events will be sent. #### Possible Answer #2: Conversion windows and data sync GrowthBook may exclude some conversions if they fall outside the metric conversion window, or there may be delays or issues with data synchronization between GA4 and GrowthBook. If you have checked your implementation and still see fewer events, we recommended logging the execution of the `trackingCallback` and verifying that `window.gtag` is indeed available and being called as expected. Also, check for any consent or ad blocker issues that might prevent events from reaching GA4. ## Use GA4's built-in debugging tools Check the Real-Time reports in GA4 for event validation. ## Use GrowthBook's Developer Tools extension to debug GA4 issues GrowthBook's [DevTools Browser Extension](/tools/chrome-extension) (available for [Chrome](https://chromewebstore.google.com/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/)) is very helpful for debugging issues with GA4 by: - Inspecting whether the GrowthBook SDK is properly loaded on the web site - Checking which feature flags and experiments are active - Verifying that user attributes are set as expected The extension allows you to see real-time information about feature flags and experiments, which helps to confirm if the SDK is initialized correctly and if experiments are being evaluated as intended. This is especially useful when troubleshooting why experiment exposure events (like `experiment_viewed`) might not be appearing in GA4. ### Initial debugging steps - Check if the SDK is detected (look for a green dot on the extension icon). - Inspect which experiments and feature flags are active and their current values. - Verify that user attributes (such as IDs used for bucketing) are set correctly. - If you’re not seeing expected experiments or flags, this may indicate issues with SDK initialization, attribute configuration, or experiment targeting, which could explain missing events in GA4. ## Use your browser's Developer Tools Console to debug GTM/GA4 issues If you've confirmed that the `trackingCallback` is implemented and firing, GA4 is loaded, and you are still not seeing `experiment_viewed` events, we recommend: - Check for JavaScript errors in the browser's Console that are related to your feature flags. - Add logging statements throughout the GrowthBook-related code in your application, particularly in and around the SDK's `trackingCallback`, then check in your browser's Developer Tools Console to further debug the issue. - Monitor the `dataLayer` object in the browser console by typing in `dataLayer` and pressing Enter. Verify the feature flag names match what is shown in the GrowthBook dashboard. --- ## Google BigQuery (BQ) Troubleshooting # GrowthBook and Google BigQuery (BQ) Google BigQuery (BQ) is a popular data warehouse chosen by GrowthBook users because it is designed to work seamlessly with Google Analytics (GA4), and GA4 is a popular Data Source for GrowthBook integrations. This article helps troubleshoot common issues encountered when configuring BigQuery with GrowthBook and can be used in combination with our guides on [Configuring GrowthBook to Work with BigQuery](/guide/bigquery) and [A/B Testing with Google Analytics 4 (GA4) and GrowthBook](/guide/GA4-google-analytics). Please watch the [How GrowthBook Works with Your Data](https://youtu.be/hboLeUbpdus) (4 minutes) overview video to better understand the issues and solutions in this troubleshooting guide. ## Why do I not see any experiment data in BigQuery? The most common issue reported by GrowthBook users in relation to BigQuery is not seeing any data being pulled into BigQuery from GA4. ### Initial sync delay for new GA4 properties When you first link your GA4 property to BigQuery, there is typically a 24–48 hour delay before the first daily export starts showing up in your BigQuery dataset. This is a one-time setup delay imposed by Google, not GrowthBook, and is expected behavior. If 48 hours have passed since a new experiment was started and there is still no experiment data in BigQuery, you'll need to ensure the connection between GA4 and BigQuery was configured properly. Notably, GrowthBook does not send experiment data directly to BigQuery. GrowthBook first sends the experiment data to your Data Source (usually GA4 if you're using BigQuery) via the `trackingCallback`, and from there the data is synced to BigQuery from GA4 every 24 hours (see below). ## Why is recent experiment data not yet available in BigQuery? The standard (free) tier of GA4 sends a daily export to BigQuery for the previous day's data. This means that there's always about a 24-hour delay between when data is collected in GA4 and when it becomes available in BigQuery. GA4 offers a streaming export option that sends event data to BigQuery within minutes, rather than once per day, but this is only available for GA4 properties on Google Analytics 360 which is the paid Enterprise version. --- ## Google Tag Manager (GTM) Troubleshooting # GrowthBook and Google Tag Manager (GTM) Google Tag Manager (GTM) is commonly configured by GrowthBook users to work alongside their Google Analytics (GA4) instance. This article helps troubleshoot common issues encountered when integrating GTM directly with GrowthBook, and when integrating GTM indirectly with GrowthBook through its connection to GA4. It can be used in combination with our [Google Tag Manager Setup Guide](/guide/google-tag-manager-and-growthbook) and our guide on [A/B Testing with Google Analytics 4 (GA4) and GrowthBook](/guide/GA4-google-analytics). Please watch the [How GrowthBook Works with Your Data](https://youtu.be/hboLeUbpdus) (4 minutes) overview video to better understand the issues and solutions in this troubleshooting guide. ## Special note for the HTML and JavaScript SDKs ### GA4 and GrowthBook's HTML Script Tag SDK GrowthBook's HTML Script Tag SDK provides [automatic experiment tracking](/lib/script-tag#tracking-experiment-views) if it detects Google Tag Manager (GTM) and/or Google Analytics (GA4) on the webpage. For each experiment view, the HTML SDK sends an event with the `experiment_id` and any `variation_id`s to these analytics tools if they are present on the webpage. No additional configuration is needed for these integrations; simply include the HTML script tag as shown in the documentation and the `experiment_viewed` events will be sent automatically. If you need to define a custom `trackingCallback` you can do so with the `window.growthbook_config.trackingCallback` object. This must be set before loading the script tag. ### GA4 and GrowthBook's JavaScript SDK Similar to the details above regarding the HTML SDK, the JavaScript SDK can accommodate [an optional plugin](/lib/js#third-party-tracking) that will automatically send `experiment_viewed` events to GTM and/or GA4 if configured. The plugin is **not** enabled by default and must be imported and then added to the `plugins` array when constructing the `growthbook` instance. ## Common issues when integrating GTM with GrowthBook The issues described below are commonly reported and can lead to missing or incomplete experiment data in GA4 when using GTM with GrowthBook. ### No `experiment_viewed` events in GTM Ensure the `trackingCallback` is correctly implemented and that the required `dataLayer` variables and triggers in GTM have been configured properly by reviewing [Using GrowthBook with Google Tag Manager (GTM)](/guide/google-tag-manager-and-growthbook#4-tracking-via-dataLayer-and-gtm). If you are using the HTML Script SDK, the `experiment_viewed` event is pushed to the `dataLayer` automatically. For other SDKs, the `trackingCallback` needs to be implemented manually to ensure the `experiment_viewed` event is pushed to the `dataLayer`. ### GTM variables, triggers, or tags are not properly configured Users often miss creating the necessary `dataLayer` variables for `experiment_id` and `variation_id`, or do not set up the Custom Event Trigger for `experiment_viewed` correctly. If the GA4 Event Tag is not configured to use these variables, the event will not be sent to GA4 from GTM. Even if the event is in the `dataLayer` and the trigger fires, if the GA4 Event Tag is not configured with the correct parameters (`experiment_id`, `variation_id`), the data may not be sent or may be sent incorrectly. If the GA4 Event Tag in GTM is not set to fire on the `experiment_viewed` event, or if the trigger conditions are not met, the event will not be sent to GA4. Double-check the trigger configuration and ensure it matches the event name and parameters. ### The GA4 Measurement ID is incorrect The GA4 Measurement ID is a unique identifier for your GA4 property that's used to send data from your website or app to the correct GA4 account. It looks like "G-XXXXXXXXXX" and is required when configuring GA4 tags in Google Tag Manager or when sending events directly to GA4. You can see how the Measurement ID is used in the [GrowthBook and GTM integration](/guide/google-tag-manager-and-growthbook#4-tracking-via-datalayer-and-gtm) documentation, where it is entered as a configuration value for sending experiment events to GA4. ### Missing or incorrect `dataLayer` push If the GrowthBook SDK’s `trackingCallback` does not push the `experiment_viewed` event to the `dataLayer`, or if the event structure is incorrect, GTM will not detect or forward the event to GA4. The event must be pushed in the correct format, such as: ```js // JavaScript window.dataLayer = window.dataLayer || []; window.dataLayer.push({ event: "experiment_viewed", experiment_id: experiment.key, variation_id: result.key, }); ``` See the [Tracking via `dataLayer` and GTM](/guide/google-tag-manager-and-growthbook#4-tracking-via-dataLayer-and-gtm) documentation for more information. ### Missing hash attribute The [hash attribute](/faq#do-users-always-get-assigned-the-same-experiment-variation) is a unique identifier for a user, such as a `user_id` or a `device_id`. It's a crucial part of the GrowthBook SDK because it is used to determine which users are included in an experiment. If the hash attribute is missing, it can prevent the `trackingCallback` from firing. Ensure that the hash attribute is correctly set when initializing the GrowthBook instance. GrowthBook's [DevTools Browser Extension](/tools/chrome-extension) (available for [Chrome](https://chromewebstore.google.com/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/)) can help debug issues with the hash attribute. ### Sequencing and timing issues GrowthBook's [DevTools Browser Extension](/tools/chrome-extension) (available for [Chrome](https://chromewebstore.google.com/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/)) is especially helpful for troubleshooting timing and sequencing issues common with GTM setups because it lets you confirm whether GrowthBook is running and receiving the correct data on your site. #### GTM not loaded before GrowthBook's `trackingCallback` is sent If GrowthBook fires the `experiment_viewed` event via the `trackingCallback` before GTM is fully loaded, the event may be missed. This can happen if scripts are loaded in the wrong order or if there are delays in page load. #### Loading GrowthBook via GTM (not recommended) Loading GrowthBook via GTM can introduce delays, and if GTM is slow to load, the event may be pushed to the `dataLayer` before GTM is ready to listen for it. #### Avoiding sequencing and timing issues Ensure that the GrowthBook SDK is loaded and initialized only after GTM has finished loading. One recommended approach is to add the GrowthBook script directly to the page’s `` instead of injecting it via GTM, as this can help with performance and sequencing issues. #### Troubleshoot using GTM's preview mode Additionally, you can use GTM’s preview mode to verify the order in which tags are firing and adjust your triggers or tag sequencing to ensure GTM is ready before GrowthBook’s `trackingCallback` executes. This helps ensure that the `experiment_viewed` event is captured reliably by GTM and forwarded to GA4. ### Consent and cookie banner blocking If a cookie consent banner blocks analytics until consent is given, the `experiment_viewed` event may be sent before consent, causing it to be dropped. Delay initializing the GrowthBook SDK and firing the `trackingCallback` (which will send the `experiment_viewed` event) until after the user has granted consent. This ensures that analytics and experiment exposure events are only sent when it is legally and technically permitted. #### Special instructions if using the HTML Script Tag SDK or the Edge SDK Read more about cookies and GDPR compliance in the docs for the [HTML Script Tag SDK](/lib/script-tag#cookie-policy-and-gdpr) and [Edge SDK](/lib/edge/lambda#cookie-policy-and-gdpr). #### Persist the cookie after consent Once consent is given, you can manually persist the cookie by dispatching the following event: ```js // JavaScript document.dispatchEvent(new CustomEvent("growthbookpersist")); ``` Read more about cookies and GDPR compliance in the docs for the [HTML Script Tag SDK](/lib/script-tag#cookie-policy-and-gdpr) and [Edge SDK](/lib/edge/lambda#cookie-policy-and-gdpr). ### Ad Blockers or Privacy Extensions Some users may have ad blockers or privacy browser extensions that block analytics scripts or `dataLayer` pushes, ultimately leading to missing events in GA4. In these cases, the GTM container may not load at all, no tags fire, and/or GA4 events may never be sent, even if GTM loads. Thus conversion tracking, user behavior data, and pageviews are underreported ### Slow web page loading times and latency If the site loads slowly, users may navigate away before the `experiment_viewed` event is pushed to the `dataLayer` and picked up by GTM, resulting in lost events. ### GTM Security Restrictions If you are injecting the GrowthBook script via GTM (not recommended), GTM may strip out data attributes (like `data-client-key`), causing the SDK to not initialize properly and thus not push events to the `dataLayer`. ### Data Propagation Delay Sometimes, there is a delay between when the event is sent and when it appears in GA4. This is normal, but if you never see the event, it’s likely a configuration issue. ## Use GTM's Preview Mode to debug If you have verified your GTM setup and still see discrepancies, use GTM's Preview mode to ensure the `experiment_viewed` event is being pushed to the `dataLayer` and that your GA4 Event Tag is firing as expected, including in the correct tag firing sequence. Also, check GA4's real-time reports to confirm event receipt ([more details in our GTM setup guide](/guide/google-tag-manager-and-growthbook#testing-your-setup)). ## Use GA4's DebugView (if applicable) Use GA4's DebugView for real-time event validation if you are using GA4 in combination with GTM. ## Use GrowthBook's Developer Tools extension to debug GrowthBook's [DevTools Browser Extension](/tools/chrome-extension) (available for [Chrome](https://chromewebstore.google.com/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/)) is very helpful for debugging issues with GTM (and GA4) by: - Inspecting whether the GrowthBook SDK is properly loaded on the web site - Checking which feature flags and experiments are active - Verifying that user attributes are set as expected The extension allows you to see real-time information about feature flags and experiments, which helps to confirm if the SDK is initialized correctly and if experiments are being evaluated as intended. This is especially useful when troubleshooting why experiment exposure events (like `experiment_viewed`) might not be appearing in GTM/GA4. ### Initial debugging steps - Check if the SDK is detected (look for a green dot on the extension icon). - Inspect which experiments and feature flags are active and their current values. - Verify that user attributes (such as IDs used for bucketing) are set correctly. - If you’re not seeing expected experiments or flags, this may indicate issues with SDK initialization, attribute configuration, or experiment targeting, which could explain missing events in GTM/GA4. ## Use your browser's Developer Tools Console to debug GTM/GA4 issues If you've confirmed that the `trackingCallback` is implemented and firing, GA4 is loaded, and you are still not seeing `experiment_viewed` events, we recommend: - Check for JavaScript errors in the browser's Console that are related to your feature flags. - Add logging statements throughout the GrowthBook-related code in your application, particularly in and around the SDK's `trackingCallback`, then check in your browser's Developer Tools Console to further debug the issue. - Monitor the `dataLayer` object in the browser console by typing in `dataLayer` and pressing Enter. Verify the feature flag names match what is shown in the GrowthBook dashboard. --- ## Savings from Fact Metric Optimization import MaxWidthImage from "@site/src/components/MaxWidthImage"; Fact Tables can dramatically reduce the number of rows scanned by an experiment analysis in GrowthBook. This article explains how Fact Tables work and how they can optimize your queries. ## How do Fact Tables Optimize Queries? [Fact Tables](/app/metrics) are one of the best tools for controlling query runtime and costs in GrowthBook for Enterprise customers. You can read more about how we achieve the optimization [in this blog post](https://blog.growthbook.io/fact-table-query-optimization/), but the following section provides an overview. Fact Tables allow you to define multiple metrics on top of one shared table in your data warehouse. When multiple metrics from the same Fact Table are added to an experiment, we only scan that Fact Table once for all metrics when updating results. Therefore, if multiple metrics come from the same table in your warehouse, you should define them in a single Fact Table. Here's the analyses we run **without fact tables**. As you can see, each experiment assignment source is scanned and joined to the metric source for each experiment. Furthermore, each metric is scanned separately, even if they come from the same underlying table in your data warehouse. ![Analysis Without Fact Tables](/images/metrics/fact-metric-before-optimization.png) **With fact tables**, we only need to scan the experiment assignment source once per Fact Table and join it once to a single scan of the Fact Table, even if you have 10 metrics defined on that fact table. We then store the metrics in a wide format in the join User Metric table and then aggregate them all in one query. ![Analysis With Fact Tables](/images/metrics/fact-metric-after-optimization.png) ## How Many Fewer Rows are Scanned? Imagine you have 10 metrics that come from 2 tables and each table has around 100M rows. Furthermore, say your experiment assignment source has around 10M rows itself. For an analysis without Fact Tables, you would scan (10M\*10 + 100M\*10)=1,100M rows. For an analysis where the 10 metrics are in 2 Fact Tables, you would scan (100M\*2 + 10M\*2)=220M rows. That is a reduction from 1,100M to 220M rows scanned! With BigQuery on-demand pricing, this represents huge savings per analysis. Over the course of an experiment that updates daily, these savings compound. Furthermore, because we place the data from a Fact Table in a wide table, we do far less computationally costly joins with Fact Tables. This means that Fact Tables will also often reduce costs associated with compute or runtime, not just rows scanned. In general the savings go up when you **increase the number of metrics per fact table** and as you **increase the number of metrics per experiment**. ## Recommendations If you have multiple metrics that are defined on the same table (e.g. a “sign-ups” table or a very broad “events” table), then it makes sense to create a Fact Table for that entire table and use Filters and Column definitions to help build your Metrics Library. The more metrics you can define on one table, the better, but sometimes it can make sense to create two Fact Tables to help keep your Metrics library organized or to prevent having to do JOIN statements directly in your Fact Table SQL. Read more about [Fact Tables in our docs](/app/metrics). --- ## Build Your Own # Build Your Own SDK **Latest spec version: 0.7.1 [View Changelog](#changelog)** This guide is meant for library authors looking to build a GrowthBook SDK in a currently unsupported language. All libraries should follow this specification as closely as the language permits to maintain consistency and make updates and maintenance easier. At the end of this guide is an extensive test suite with over 400 language-agnostic unit tests in JSON format. All SDKs must pass 100% of these test cases. It's recommended to add additional manual unit tests as needed on top of these. ## Data structures Here are a number of important data structures in GrowthBook SDKs, listed alphabetically. ### Attributes **Attributes** are an arbitrary JSON object containing user and request attributes. Here's an example: ```json { "id": "123", "anonId": "abcdef", "company": "growthbook", "url": "/pricing", "country": "US", "browser": "firefox", "age": 25, "beta": true, "account": { "plan": "team", "seats": 10 } } ``` ### BucketRange A tuple that describes a range of the numberline between `0` and `1`. The tuple has 2 parts, both floats - the start of the range and the end. For example: ```ts [0.3, 0.7]; ``` ### Condition A **Condition** is evaluated against **Attributes** and used to target features/experiments to specific users. The syntax is inspired by MongoDB queries. Here is an example: ```ts { "country": "US", "browser": { "$in": ["firefox", "chrome"] }, "email": { "$not": { "$regex": "@gmail.com$" } } } ``` ### ParentCondition A **ParentCondition** defines a prerequisite. It consists of a parent feature's **id** (`string`), a **condition** (`Condition`), and an optional **gate** (`boolean`) flag. Instead of evaluating against attributes, the condition evaluates against the returned value of the parent feature. The condition will always reference a "value" property. Here is an example of a gating prerequisite where the parent feature must be on (`true`) ``` { "id": "parent-feature", "condition": { "value": { "$eq": true } }, "gate": true } ``` ### Context **Context** object passed into the GrowthBook constructor. Has a number of optional properties: - **enabled** (`boolean`) - Switch to globally disable all experiments. Default true. - **apiHost** (`string`) - The GrowthBook API Host. Optional - **clientKey** (`string`) - The key used to fetch features from the GrowthBook API. Optional - **decryptionKey** (`string`) - The key used to decrypt encrypted features from the API. Optional - **attributes** (`Attributes`) - Map of user attributes that are used to assign variations - **url** (`string`) - The URL of the current page - **features** (`FeatureMap`) - Feature definitions (usually pulled from an API or cache) - **forcedVariations** (`ForcedVariationsMap`) - Force specific experiments to always assign a specific variation (used for QA) - **qaMode** (`boolean`) - If true, random assignment is disabled and only explicitly forced variations are used. - **trackingCallback** (`TrackingCallback`) - A function that takes `experiment` and `result` as arguments. ### Experiment Defines a single **Experiment**. Has a number of properties: - **key** (`string`) - The globally unique identifier for the experiment - **variations** (`any[]`) - The different variations to choose between - **weights** (`float[]`) - How to weight traffic between variations. Must add to 1. - **active** (`boolean`) - If set to false, always return the control (first variation) - **coverage** (`float`) - What percent of users should be included in the experiment (between 0 and 1, inclusive) - **ranges** (`BucketRange[]`) - Array of ranges, one per variation - **condition** (`Condition`) - Optional targeting condition - **namespace** (`Namespace`) - Adds the experiment to a namespace - **force** (`integer`) - All users included in the experiment will be forced into the specific variation index - **hashAttribute** (`string`) - What user attribute should be used to assign variations (defaults to `id`) - **fallbackAttribute** (`string`) - When using sticky bucketing, can be used as a fallback to assign variations - **hashVersion** (`integer`) - The hash version to use (default to `1`) - **meta** (`VariationMeta[]`) - Meta info about the variations - **filters** (`Filter[]`) - Array of filters to apply - **seed** (`string`) - The hash seed to use - **name** (`string`) - Human-readable name for the experiment - **phase** (`string`) - Id of the current experiment phase - **disableStickyBucketing** (`boolean`) - If true, sticky bucketing will be disabled for this experiment. (Note: sticky bucketing is only available if a StickyBucketingService is provided in the Context) - **bucketVersion** (`integer`) - An sticky bucket version number that can be used to force a re-bucketing of users (default to `0`) - **minBucketVersion** (`integer`) - Any users with a sticky bucket version less than this will be excluded from the experiment The only required properties are `key` and `variations`. Everything else is optional. ### ExperimentResult The result of running an **Experiment** given a specific **Context** - **inExperiment** (`boolean`) - Whether or not the user is part of the experiment - **variationId** (`int`) - The array index of the assigned variation - **value** (`any`) - The array value of the assigned variation - **hashUsed** (`boolean`) - If a hash was used to assign a variation - **hashAttribute** (`string`) - The user attribute used to assign a variation - **hashValue** (`string`) - The value of that attribute - **featureId** (`string` or `null`) - The id of the feature (if any) that the experiment came from - **key** (`string`) - The unique key for the assigned variation - **bucket** (`float`) - The hash value used to assign a variation (float from `0` to `1`) - **name** (`string` or `null`) - The human-readable name of the assigned variation - **passthrough** (`boolean`) - Used for holdout groups - **stickyBucketUsed** (`boolean`) - If sticky bucketing was used to assign a variation The `variationId` and `value` should always be set, even when `inExperiment` is false. The `hashAttribute` and `hashValue` should always be set, even when `hashUsed` is false. The `key` should always be set, even if `experiment.meta` is not defined or incomplete. In that case, convert the variation's array index to a string (e.g. `0` -> `"0"`) and use that as the `key` instead. ### Feature A **Feature** object consists of a default value plus rules that can override the default. - **defaultValue** (`any`) - The default value (should use `null` if not specified) - **rules** (`FeatureRule[]`) - Array of **FeatureRule** objects that determine when and how the defaultValue gets overridden ### FeatureMap A hash or map of **Feature** objects. Keys are string ids for the features. Values are **Feature** objects. For example: ```js { "feature-1": { "defaultValue": false }, "my_other_feature": { "defaultValue": 1, "rules": [ { "force": 2 } ] } } ``` ### FeatureResult The result of evaluating a **Feature**. Has a number of properties: - **value** (`any`) - The assigned value of the feature - **on** (`boolean`) - The assigned value cast to a boolean - **off** (`boolean`) - The assigned value cast to a boolean and then negated - **source** (`enum`) - One of "unknownFeature", "defaultValue", "force", "experiment", "override", "prerequisite", "cyclicPrerequisite" - **experiment** (`Experiment` or `null`) - When source is "experiment", this will be an Experiment object - **experimentResult** (`ExperimentResult` or `null`) - When source is "experiment", this will be an ExperimentResult object - **ruleId** (`string`) - The id of the feature rule that the assigned value came from (empty string if no rule was used) ### FeatureRule Overrides the defaultValue of a **Feature**. Has a number of optional properties - **id** (`string`) - Optional rule id. Default to empty string if missing. - **condition** (`Condition`) - Optional targeting condition - **parentConditions** (`ParentCondition[]`) - Each item defines a prerequisite where a `condition` must evaluate against a parent feature's value (identified by `id`). If `gate` is true, then this is a blocking feature-level prerequisite; otherwise it applies to the current rule only. - **coverage** (`float`) - What percent of users should be included in the experiment (between 0 and 1, inclusive) - **force** (`any`) - Immediately force a specific value (ignore every other option besides condition and coverage) - **variations** (`any[]`) - Run an experiment (A/B test) and randomly choose between these variations - **key** (`string`) - The globally unique tracking key for the experiment (default to the feature key) - **weights** (`float[]`) - How to weight traffic between variations. Must add to 1. - **namespace** (`Namespace`) - Adds the experiment to a namespace - **hashAttribute** (`string`) - What user attribute should be used to assign variations (defaults to `id`) - **hashVersion** (`integer`) - The hash version to use (default to `1`) - **range** (`BucketRange`) - A more precise version of `coverage` - **ranges** (`BucketRange[]`) - Ranges for experiment variations - **meta** (`VariationMeta[]`) - Meta info about the experiment variations - **filters** (`Filter[]`) - Array of filters to apply to the rule - **seed** (`string`) - Seed to use for hashing - **name** (`string`) - Human-readable name for the experiment - **phase** (`string`) - The phase id of the experiment - **tracks** (`TrackData[]`) - Array of tracking calls to fire ### Filter Object used for mutual exclusion and filtering users out of experiments based on random hashes. Has the following properties: - **seed** (`string`) - The seed used in the hash - **ranges** (`BucketRange[]`) - Array of ranges that are included - **hashVersion** (`integer`) - The hash version to use (default to `2`) - **attribute** (`string`, optional) - The attribute to use (default to `"id"`) ### ForcedVariationsMap A hash or map that forces an **Experiment** to always assign a specific variation. Useful for QA. Keys are the experiment key, values are the array index of the variation. For example: ```json { "my-test": 0, "other-test": 1 } ``` ### Namespace A tuple that specifies what part of a namespace an experiment includes. If two experiments are in the same namespace and their ranges don't overlap, they wil be mutually exclusive. The tuple has 3 parts: 1. The namespace id (`string`) 2. The beginning of the range (`float`, between `0` and `1`) 3. The end of the range (`float`, between `0` and `1`) For example: ```ts ["namespace1", 0, 0.5]; ``` ### TrackingCallback A callback function that is executed every time a user is included in an **Experiment**. Here's an example: ```js function track(experiment, result) { analytics.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.variationId, }); } ``` ### TrackData Used for remote feature evaluation to trigger the `TrackingCallback`. An object with 2 properties: - **experiment** - `Experiment` - **result** - `ExperimentResult` ### VariationMeta Meta info about an experiment variation. Has the following properties: - **key** (`string`, optional) - A unique key for this variation - **name** (`string`, optional) - A human-readable name for this variation - **passthrough** (`boolean`, optional) - Used to implement holdout groups ## Helper Functions There are some helper functions which are used a few times throughout the SDK. ### hash(seed: string, value: string, version: integer): float|null Hashes a string to a float between 0 and 1. Uses the simple [Fowler–Noll–Vo](https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) algorithm, specifically fnv32a. An implementation of this is available in most languages already, and if not it's only a few lines of code to implement yourself. Fnv32a returns an integer, so we convert that to a float using a modulus. The original hash version (1) had a flaw that caused bias when running experiments in parallel. ```ts // New hashing algorithm if (version === 2) { n = fnv32a(fnv32a(seed + value) + ""); return (n % 10000) / 10000; } // Original hashing algorithm (with a bias flaw) else if (version === 1) { n = fnv32a(value + seed); return (n % 1000) / 1000; } return null; ``` **Note**: It's important to use the exact hashing algorithms outlined here so all SDKs behave identically. ### inRange(n: float, range: BucketRange): boolean Determines if a number `n` is within the provided range. ```js return n >= range[0] && n ; ``` ### inNamespace(userId: string, namespace: Namespace): boolean This checks if a userId is within an experiment namespace or not. The `namespace` argument is a tuple with 3 parts: id (string), start (float), and end (float). 1. Hash the userId and namespace name with two underscores as a delimiter ```ts n = hash("__" + namespace[0], userId, 1); ``` 2. Return if hash is greater than (inclusive) the namespace start and less than (exclusive) the namespace end: ```ts return n >= namespace[1] && n 1) coverage = 1; ``` 2. Default to equal weights if the weights don't match the number of variations. ```ts if (weights.length != numVariations) { weights = getEqualWeights(numVariations); } ``` 3. Default to equal weights if the sum is not equal `1` (or close enough when rounding errors are factored in): ```ts if (sum(weights) 1.01) { weights = getEqualWeights(numVariations); } ``` 4. Convert weights to ranges and return ```ts cumulative = 0; ranges = []; for (w in weights) { start = cumulative; cumulative += w; ranges.push([start, start + coverage * w]); } return ranges; ``` Some examples: - `getBucketRanges(2, 1, [0.5, 0.5])` -> `[[0, 0.5], [0.5, 1]]` - `getBucketRanges(2, 0.5, [0.4, 0.6])` -> `[[0, 0.2], [0.4, 0.7]]` ### chooseVariation(n: float, ranges: BucketRange[]): integer Given a hash and bucket ranges, assign one of the bucket ranges. 1. Loop through ranges 1. If n is within the range, return the range index ```ts if (inRange(n, ranges[i])) { return i; } ``` 2. Return `-1` if it makes it through the whole ranges array without returning If multiple ranges match, return the first matching one. ### getQueryStringOverride(id: string, url: string, numVariations: integer): null|integer This checks if an experiment variation is being forced via a URL query string. This may not be applicable for all SDKs (e.g. mobile). As an example, if the id is `my-test` and url is `http://localhost/?my-test=1`, you would return `1`. If possible, you should use a proper URL parsing library vs relying on simple regexes. Return `null` if any of these are true: - There is no querystring - The id is not a key in the querystring - The variation is not an integer - The variation is less than 0 or greater than or equal to numVariations ### decrypt(encryptedString: string, decryptionKey: string): string This decrypts a string using the AES-CBC 128KB algorithm. This is used if the GrowthBook App is configured to encrypt feature flag definitions. Here's an example in PHP: ```php function decrypt(string $encryptedString, string $decryptionKey) { // Split the string into two parts, delimited by "." list($iv, $cipherText) = explode(".", $encryptedString, 2); // The Initialization Vector (iv) is base64 encoded $iv = base64_decode($iv); // Decrypt using the AES-CBC 128kb algorithm // Will throw an Exception if unable to decrypt return openssl_decrypt($cipherText, "aes-128-cbc", $decryptionKey, 0, $iv); } ``` The return value will be a JSON-encoded string. If an error occurs, you can throw an exception (or whatever is typically used for error handling). ## Evaluating Conditions In addition to the helper functions above, there are a number of methods related to evaluating targeting conditions. There is only one public method `evalCondition` and everything else is a private helper function. ### public evalCondition(attributes: Attributes, condition: Condition): boolean This is the main function used to evaluate a condition. It loops through the condition key/value pairs and checks each entry: 1. If condition key is `$or`, check if `evalOr(attributes, condition["$or"])` is false. If so, break out of the loop and return false 2. If condition key is `$nor`, check if `!evalOr(attributes, condition["$nor"])` is false. If so, break out of the loop and return false 3. If condition key is `$and`, check if `evalAnd(attributes, condition["$and"])` is false. If so, break out of the loop and return false 4. If condition key is `$not`, check if `!evalCondition(attributes, condition["$not"])` is false. If so, break out of the loop and return false 5. Otherwise, check if `evalConditionValue(value, getPath(attributes, key))` is false. If so, break out of the loop and return false If none of the entries failed their checks, `evalCondition` returns true ### private evalOr(attributes: Attributes, conditions: Condition[]): boolean `conditions` is an array of Condition objects 1. If conditions is empty, return true 2. Loop through conditions 1. If `evalCondition(attributes, conditions[i])` is true, break out of the loop and return true 3. Return false ### private evalAnd(attributes: Attributes, conditions: Condition[]): boolean `conditions` is an array of Condition objects 1. Loop through conditions 1. If `evalCondition(attributes, conditions[i])` is false, break out of the loop and return false 2. Return true ### private isOperatorObject(obj): boolean This accepts a parsed JSON object as input and returns `true` if every key in the object starts with `$`. - `{"$gt": 1}` -> `true` - `{"$gt": 1, "$lt": 10}` -> `true` - `{"foo": "bar"}` -> `false` - `{"$gt": 1, "foo": "bar"}` -> `false` If the object is empty and has no keys, this should also return true. ### private getType(attributeValue): string This returns the data type of the passed in argument. The valid types to return are: - `string` - `number` - `boolean` - `array` - `object` - `null` - `undefined` - `unknown` The difference between `null` and `undefined` can be illustrated as follows: ```ts obj = JSON.parse('{"foo": null}'); getType(obj["foo"]); // null getType(obj["bar"]); // undefined ``` The value `unknown` is there just in case you can't figure out the data type for whatever reason. It will never be used in most implementations. ### private getPath(attributes: Attributes, path: string): any Given attributes and a dot-separated path string, return the value at that path (or `null`/`undefined` if the path doesn't exist) Given the input: ```json { "name": "john", "job": { "title": "developer" } } ``` It should return: - `getPath(input, "name")` -> `"john"` - `getPath(input, "job.title")` -> `"developer"` - `getPath(input, "job.company")` -> `null` or `undefined` ### private evalConditionValue(conditionValue, attributeValue): boolean 1. If `conditionValue` is an object and `isOperatorObject(conditionValue)` is true 1. Loop over each key/value pair 1. If `evalOperatorCondition(key, attributeValue, value)` is false, return false 2. Return true 2. Else, do a deep comparison between `attributeValue` and `conditionValue`. Return true if equal, false if not. ### private elemMatch(conditionValue, attributeValue): boolean This checks if `attributeValue` is an array, and if so at least one of the array items must match the condition 1. If `attributeValue` is not an array, return false 2. Loop through items in `attributeValue` 1. If `isOperatorObject(conditionValue)` 1. If `evalConditionValue(conditionValue, item)`, break out of loop and return true 2. Else if `evalCondition(item, conditionValue)`, break out of loop and return true 3. Return false ### private paddedVersionString(input): string This function can be used to help with the evaluation of the version string comparison. There are 6 operators that are used for comparing version strings, e.g. `v1.2.3` or `1.2.3`: | Condition | Comparison | Description | | --------- | ---------- | ---------------------------------------------------------------- | | `$veq` | `==` | Versions are equal | | `$vne` | `!=` | Versions are not equal | | `$vlt` | `` | The first version is greater than the second version | | `$vgte` | `>=` | The first version is greater than or equal to the second version | Rules: - Segments are separated by `.` and `-` characters - Segments should be compared alphanumerically from the left-most segment - Digit-only segments should be left-padded with a space so that they have the same number of characters. - A leading `v` in a version string should be ignored - Semantic version syntax used to denote build information (as denoted by a `+`, e.g. `+mybuild`) should be ignored for comparisons. Here's an example: ```ts export function paddedVersionString(input: string): string { // Remove build info and leading `v` if any // Split version into parts (both core version numbers and pre-release tags) // "v1.2.3-rc.1+build123" -> ["1","2","3","rc","1"] const parts = input.replace(/(^v|\+.*$)/g, "").split(/[-.]/); // If it's SemVer without a pre-release, add `~` to the end // ["1","0","0"] -> ["1","0","0","~"] // "~" is the largest ASCII character, so this will make "1.0.0" greater than "1.0.0-beta" for example if (parts.length === 3) { parts.push("~"); } // Left pad each numeric part with spaces so string comparisons will work ("9">"10", but " 9" (v.match(/^[0-9]+$/) ? v.padStart(5, " ") : v)) .join("-"); } ``` ### private isIn(conditionValue, actualValue): boolean Checks to see if `actualValue` is in the `conditionValue` array. This implements the `$in` and `$nin` operators. 1. If `actualValue` is an array 1. Return `true` if the intersection between `actualValue` and `conditionValue` has at least 1 element. Otherwise, return `false`. 2. Else 1. Return `true` if `conditionValue` contains `actualValue`. Otherwise, return `false`. ### private evalOperatorCondition(operator, attributeValue, conditionValue) This function is just a case statement that handles all the possible operators There are basic comparison operators in the form `attributeValue {op} conditionValue`: - `$eq` == - `$ne` != (not equals) - `$lt` < - `$lte` <= - `$gt` > - `$gte` >= - `$regex` ~ (regex match) There are 3 operators where conditionValue is an array. All of these should return `false` if `conditionValue` is not an array for whatever reason. - `$in` 1. Return `isIn(conditionValue, attributeValue)` - `$nin` 1. Return `not isIn(conditionValue, attributeValue)` - `$all` 1. If attributeValue is not an array, return `false` 2. Loop through conditionValue array 1. If none of the elements in the attributeValue array pass `evalConditionValue(conditionValue[i], attributeValue[j])`, return false 3. Return true There are 2 operators where attributeValue is an array: - `$elemMatch` 1. Return `elemMatch(conditionValue, attributeValue)` - `$size` 1. If attributeValue is not an array, return false 2. Return `evalConditionValue(conditionValue, attributeValue.length)` There are 3 other operators: - `$exists` 1. If conditionValue is false, return true if attributeValue is null or undefined 2. Else, return true if attributeValue is NOT null or undefined 3. Return false by default - `$type` 1. Return `getType(attributeValue) == conditionValue` - `$not` 1. Return `!evalConditionValue(conditionValue, attributeValue)` There are 6 operators that are used for comparing version strings, e.g. `v1.2.3` or `1.2.3`. See [paddedVersionString(input)](#private-paddedversionstringinput-string) for details. If operator doesn't match any of these, return false and potentially log the error for debug purposes. ## GrowthBook Class The GrowthBook class is the main export of the SDK. ### constructor The constructor takes a Context object and stores the properties for later. Nothing else needs to be done during initialization. This class has a few helper methods as well as 2 main public methods - `evalFeature` and `run`. ### Getters and Setters There should be simple getters and setters for a few of the context properties: - `attributes` - `features` - `forcedVariations` - `url` - `enabled` ### private getFeatureResult(value, source, experiment, experimentResult): FeatureResult This is a helper method to create a `FeatureResult` object. The first two arguments, `value`, and `source` are required. The last two, `experiment` and `experimentResult` are optional and should default to `null`. Besides the passed-in arguments, there are two derived values - `on` and `off`, which are just the value cast to booleans. value can be any JSON type. Only the following values are considered to be "falsy": - `null` - `false` - `""` - `0` Everything else is considered "truthy", including empty arrays and objects. If value is "truthy", then `on` should be true and `off` should be false. If the value is "falsy", then they should take opposite values. ### private isFilteredOut(filters: Filters[]): boolean This is a helper method to evaluate `filters` for both feature flags and experiments. 1. Loop through filters array 1. Get the hashAttribute and hashValue ```ts hashAttribute = filter.attribute || "id"; hashValue = context.attributes[hashAttribute] || ""; ``` 2. If hashValue is empty, return `true` 3. Determine the bucket for the user ```ts n = hash(filter.seed, hashValue, filter.hashVersion || 2); ``` 4. If `inRange(n, range)` is false for every range in `filter.ranges`, return `true` 2. If you made it through the entire array without returning early, return `false` now ### private isIncludedInRollout(seed: string, hashAttribute: string | null, range: BucketRange | null, coverage: float | null, hashVersion: integer | null): boolean Determines if the user is part of a gradual feature rollout. 1. Either `coverage` or `range` are required. If both are `null`, return `true` immediately 2. If `range` is null and `coverage` is zero, return `false` immediately. This catches an edge case where the bucket is zero and users are let through when they shouldn't be 3. Get the hashAttribute and hashValue ```ts hashAttribute = hashAttribute || "id"; hashValue = context.attributes[hashAttribute] || ""; ``` 4. If `hashValue` is empty, return `false` immediately 5. Determine the bucket for the user ```ts n = hash(seed, hashValue, hashVersion || 1) ``` 6. Check if user is included ```js if (range) { return inRange(n, range) } else if (coverage !== null) { return n = experiment.variations.length) { variationIndex = 0; inExperiment = false; } ``` 2. Get the hashAttribute and hashValue ```ts hashAttribute = experiment.hashAttribute || "id"; hashValue = context.attributes[hashAttribute] || ""; ``` 3. Get meta info for the assigned variation (if any) ```ts meta = experiment.meta ? experiment.meta[variationIndex] : null ``` 4. Build return object ```ts res = { key: (meta && meta.key) ? meta.key : ("" + variationIndex), featureId: featureId, inExperiment: inExperiment, hashUsed: hashUsed, variationId: variationIndex, value: experiment.variations[variationIndex], hashAttribute: hashAttribute, hashvalue: hashValue, }; ``` 5. Add optional properties and return ```ts if (meta && meta.name) res.name = meta.name; if (meta && meta.passthrough) res.passthrough = true; if (bucket !== null) res.bucket = bucket; return res; ``` ### public evalFeature(key: string): FeatureResult The `evalFeature` method takes a single string argument, which is the unique identifier for the feature and returns a `FeatureResult` object. ```js growthbook = new GrowthBook(context); myFeature = growthbook.evalFeature("my-feature"); ``` There are a few ordered steps to evaluate a feature 1. If the key doesn't exist in `context.features` 1. Return `getFeatureResult(null, "unknownFeature")` 2. Loop through the feature rules (if any) 1. If the rule has `parentConditions` (prerequisites) defined, loop through each one: 1. Call `evalFeature` on the parent condition - If a cycle is detected, break out of feature evaluation and return `getFeatureResult(null, "cyclicPrerequisite")` 2. Using the evaluated parent's result, create an object ```ts const evalObj = { "value": parentResult.value } ``` 3. Evaluate this object against the parentCondition's condition: ```ts evalCondition(evalObj, parentResult.value); ``` 4. If any of the parentConditions fail evaluation then: - If `parentCondition.gate` is true (a blocking prerequisite), return `getFeatureResult(null, "prerequisite")` - Otherwise, skip this rule and continue to the next one 2. If the rule has `filters` defined 1. if `isFilteredOut(rule.filters)`, skip this rule and continue to the next one 3. If the rule has a `condition` 1. If `evalCondition(context.attributes, rule.condition)` is false, skip this rule and continue to the next one 4. If `rule.force` is set 1. If not `isIncludedInRollout`, skip this rule and continue to the next one ```ts if (!isIncludedInRollout( rule.seed || featureKey, rule.hashAttribute, rule.range, rule.coverage, rule.hashVersion )) { continue; } ``` 2. If `rule.tracks` is set, fire the `TrackingCallback` for each element in the `rule.tracks` array. 3. Return `getFeatureResult(rule.force, "force")` 5. Otherwise, convert the rule to an Experiment object ```ts const exp = { variations: rule.variations, key: rule.key || featureKey, }; ``` 6. Copy over additional settings from the rule to `exp` if defined: - `coverage` - `weights` - `hashAttribute` - `fallbackAttribute` - `disableStickyBucketing` - `bucketVersion` - `minBucketVersion` - `namespace` - `meta` - `ranges` - `name` - `phase` - `seed` - `hashVersion` - `filters` - `condition` 7. Run the experiment ```ts result = run(exp); ``` 8. If `result.inExperiment` is false OR `result.passthrough` is true, skip this rule and continue to the next one 9. Otherwise, return ```ts return getFeatureResult(result.value, "experiment", exp, result); ``` 3. Return `getFeatureResult(feature.defaultValue || null, "defaultValue")` ### public run(experiment: Experiment): ExperimentResult The `run` method takes an Experiment object and returns an `ExperimentResult`. There are a bunch of ordered steps to run an experiment: 1. If `experiment.variations` has fewer than 2 variations, return `getExperimentResult(experiment)` 2. If `context.enabled` is false, return `getExperimentResult(experiment)` 3. If `context.url` exists ```ts qsOverride = getQueryStringOverride(experiment.key, context.url); if (qsOverride != null) { return getExperimentResult(experiment, qsOverride); } ``` 4. Return if forced via context ```ts if (experiment.key in context.forcedVariations) { return getExperimentResult( experiment, context.forcedVariations[experiment.key] ); } ``` 5. If `experiment.active` is set to false, return `getExperimentResult(experiment)` 6. Get the user hash value and return if empty ```ts hashAttribute = experiment.hashAttribute || "id"; hashValue = context.attributes[hashAttribute] || ""; if (hashValue == "") { if (experiment.fallbackAttribute && context.attributes[experiment.fallbackAttribute]) { // check if a fallbackAttribute exists (sticky bucketing) hashAttribute = experiment.fallbackAttribute; hashValue = context.attributes[hashAttribute]; } else { // no hashAttribute or fallbackAttribute, return return getExperimentResult(experiment); } } ``` 6.5 If sticky bucketing is permitted, check to see if a sticky bucket value exists. If so, skip steps 7-8. 7. Apply filters and namespace 1. If `experiment.filters` is set ```js if (isFilteredOut(experiment.filters)) { return getExperimentResult(experiment) } ``` 2. Else if `experiment.namespace` is set, return if not in range ```ts if (!inNamespace(hashValue, experiment.namespace)) { return getExperimentResult(experiment); } ``` 8. Return if any conditions are not met, return 1. If `experiment.condition` is set, return if it evaluates to false ```ts if (!evalCondition(context.attributes, experiment.condition)) { return getExperimentResult(experiment); } ``` 2. If `experiment.parentConditions` is set (prerequisites), return if any of them evaluate to false. See the corresponding logic in **evalFeature** for more details. (Note that the `gate` flag should not be set in an experiment) 3. Apply any url targeting based on experiment.urlPatterns, return if no match 9. Choose a variation 1. If a sticky bucket value exists, use it. 1. If the found sticky bucket version is blocked (doesn't exceed `experiment.minBucketVersion`), then skip enrollment 2. Else, calculate bucket ranges for the variations and choose one ```ts ranges = experiment.ranges || getBucketRanges( experiment.variations.length, experiment.coverage ?? 1, experiment.weights ?? [] ); n = hash( experiment.seed || experiment.key, hashValue, experiment.hashVersion || 1 ); assigned = chooseVariation(n, ranges); ``` 10. If assigned == `-1`, return `getExperimentResult(experiment)` 11. If experiment has a forced variation, return ```ts if ("force" in experiment) { return getExperimentResult(experiment, experiment.force); } ``` 12. If `context.qaMode`, return `getExperimentResult(experiment)` 13. Build the result object ```ts result = getExperimentResult(experiment, assigned, true, n); `` 13.5 If sticky bucketing is permitted, store the sticky bucket value ``` 14. Fire `context.trackingCallback` if set and the combination of hashAttribute, hashValue, experiment.key, and variationId has not been tracked before 15. Return `result` ### Feature helper methods There are 3 tiny helper methods that wrap `evalFeature` for a better developer experience: ```js public isOn(key) { return this.evalFeature(key).on } public isOff(key) { return this.evalFeature(key).off } public getFeatureValue(key, fallback) { value = this.evalFeature(key).value return value === null ? fallback : value } ``` For strongly typed languages, you can use generics (if supported) for `getFeatureValue` and coerce the return value to always match the data type of fallback. If generics are not supported, you can use type-specific versions of the function such as `getFeatureValueAsString`. ## Fetching and Caching Features When the Context contains a `clientKey`, the SDK should fetch and cache features automatically. If `apiHost` is not specified, default to `https://cdn.growthbook.io`. Make sure to strip and trailing slashes on user-entered hosts (e.g. `http://example.com/` becomes `http://example.com`). Features should be fetched from `{apiHost}/api/features/{clientKey}` and all errors should be handled gracefully. A network error while fetching features should never be a fatal error that stops execution. The API responses should be parsed and cached so future GrowthBook instances with the same clientKey can avoid a duplicate network request. The standard cache TTL to use is 60 seconds. For best performance, a stale-while-revalidate pattern should be used. If a cache entry is older than the TTL, return the cached value immediately and start a background process to update the cache from the API. The initial download should be intiated by a `growthbook.init()` method call. This method may take optional parameters, such as `timeout` or `skipCache`, if it makes sense. There should be an easy way for the user to wait until features finish loading. Depending on the language, this might be an event emitter, a Promise, a callback, or something similar. Use whatever method is standard for the language. ### Server-Sent Events The API response (`/api/features/{clientKey}`) may contain a response header: > x-sse-support: enabled If set to "enabled", you are able to subscribe to the API for realtime changes to feature definitions by using the [GrowthBook Proxy](/self-host/proxy). This will let you update the cache immediately when a feature changes instead of waiting for the 60s TTL to expire. The URL for subscribing to changes is `{apiHost}/sub/{clientKey}`. SDKs should not attempt to subscribe to the `/sub/` endpoint unless the header `x-sse-support: enabled` is present on the `/api/features` endpoint response. An example implementation in JavaScript is below: ```js const channel = new EventSource(`${apiHost}/sub/${clientKey}`); channel.addEventListener("features", (event) => { const data = JSON.parse(event.data); cache.set(clientKey, data.features); }) ``` Some important things to note: - The SDK should implement reconnect logic to support both client and server dropping the connection. - The response will be the same as when fetching from the `/api/features/{clientKey}` endpoint ### Encrypted Features The `/api/features/{clientKey}` endpoint can have encryption enabled (128-bit AES-CBC). When this is the case, the API response will look like this: ```js { "features": {}, "encryptedFeatures": "abcdef123456.ghijklmnop789jksdkfaksfadfasdfkahsfa" } ``` Before you can use this response, you will need to decrypt it. This requires the user to set `Context.decryptionKey` when creating the GrowthBook instance. ## Type Hinting Most languages have some sort of strong typing support, whether in the language itself or via annotations. This helps to reduce errors and is highly encouraged for SDKs. If possible, use generics to type the return value. For example, if `experiment.variations` is type `T[]`, then `result.value` should be type `T`. Or, if the fallback of `getFeatureValue` is type `string`, the return type should also be type `string`. ## Handling Errors The general rule is to be strict in development and lenient in production. You can throw exceptions in development, but someone's production app should never crash because of a call to `growthbook.evalFeature` or `growthbook.run`. For the below edge cases in production, just act as if the problematic property didn't exist and ignore errors: - `experiment.weights` is a different length from `experiment.variations` - `experiment.weights` adds up to something other than 1 - `experiment.coverage` or `feature.coverage` is greater than 1 or less than 0 - `context.trackingCallback` throws an error - URL querystring specifies an invalid variation index For the below edge cases in production, the experiment should be disabled (everyone gets assigned variation `0`): - `experiment.coverage` is less than 0 - `experiment.force` specifies an invalid variation index - `context.forcedVariations` specifies an invalid variation index - `experiment.hashAttribute` is an empty string ## Subscriptions Sometimes it's useful to be able to "subscribe" to a GrowthBook instance and be alerted every time `growthbook.run` is called. This is different from the tracking callback since it also fires when a user is _not_ included in an experiment. ```js growthbook.subscribe(function (experiment, result) { // do something }); ``` It's best to only re-fire the callbacks for an experiment if the result has changed. That means either the `inExperiment` flag has changed or the `variationId` has changed. If it makes sense for your language, this function should return an "unsubscriber". A simple callback that removes the subscription. ```js unsubscriber = growthbook.subscribe(...) unsubscriber() ``` In addition to subscriptions you may also want to expose a `growthbook.getAllResults` method that returns a map of the latest results indexed by experiment key. ## Memory Management Subscriptions and tracking calls require storing references to many objects and functions. If it makes sense for your language, libraries should provide a `growthbook.destroy` method to remove all of these references and release their memory. ## Tests We strive to have 100% test coverage for all of our SDKs. There is a language-agnostic test suite stored as a JSON file (https://github.com/growthbook/growthbook/blob/main/packages/sdk-js/test/cases.json) with more than 400 unit tests. This extensively tests all of the public methods mentioned above. The cases.json file is an object. The keys are the function being tested, and the values are arrays of test cases. The test case arrays structure is different for each function and listed below: - **evalCondition** - name of the test case (string) - condition - attributes - expected return value (boolean) - definitions for Saved Groups referenced in the test case (object of keys: ID of list -> values: array of members) - **hash** - seed (string) - value to hash (string) - hash version to use (integer) - expected result (float) - **getBucketRange** - Name of the test case (string) - Arguments array ([numVariations, coverage, weights or null]) - expected result - **feature** (evalFeature) - name of the test case (string) - context passed into GrowthBook constructor - name of the feature (string) - expected result (FeatureResult) - **run** - name of the test case (string) - context passed into GrowthBook constructor - experiment object - expected value - inExperiment (boolean) - hashUsed (boolean) - **chooseVariation** - name of the test case (string) - n (hash) - bucket ranges - expected result - **getQueryStringOverride** - name of the test case (string) - experiment key - url - numVariations - expected result - **inNamespace** - name of the test case (string) - id - namespace - expected result - **getEqualWeights** - numVariations - expected result (weights rounded to 8 decimal places) - **decrypt** - name of the test case (string) - encrypted text (string) - decryption key (string) - expected result (string or `null` if the decryption should fail) - **stickyBucket** - name of the test case (string) - context passed into GrowthBook constructor - array of preexisting sticky bucket assignment docs - name of the feature (string) - expected result - expected sticky bucket assignment docs - **urlRedirect** - name of the test case (string) - context passed into GrowthBook constructor - expected array of result objects In addition to the above, you should write custom test cases for things like event subscriptions, tracking callbacks, getters/setters, etc. that are more language-specific. ## Getting Help Join our [Slack community](https://slack.growthbook.io?ref=docs-buildyourown) if you need help or want to chat. We're also happy to hop on a call and do some pair programming. ## Attribution Open a [GitHub issue](https://github.com/growthbook/growthbook/issues) with a link to your project and we'll make sure we add it to our docs and give you proper credit for your hard work. ## Changelog - **v0.1** 2022-05-23 - Don't skip experiment rules that are forced - **v0.2** 2022-07-19 - Add `featureId` to ExperimentResult object - **v0.2.1** 2022-08-01 - Add test case for when an experiment's hashAttribute is `null` - **v0.2.2** 2022-09-08 - Add test case for when an experiment's hashAttribute is an integer - **v0.2.3** 2022-12-06 - Add test case for when an experiment's coverage is set to 0 - **v0.3.0** 2023-01-18 - New `apiHost`, `clientKey`, and `decryptionKey` Context properties - Built-in fetching and caching - Server Sent Events (SSE) support for realtime feature updates - **v0.4.0** 2023-02-24 - Changed signature of `hash` method and added multiple hashing versions - New `inRange`, `isIncludedInRollout`, and `isFilteredOut` helper methods - New `hashVersion`, `range`, `ranges`, `meta`, `filters`, `seed`, `name`, `tracks`, and `phase` properties of FeatureRules - New `hashVersion`, `ranges`, `meta`, `filters`, `seed`, `name`, and `phase` properties of Experiments - New `key`, `name`, `bucket`, and `passthrough` fields in Experiment Results - New `Filter`, `VariationMeta`, and `TrackData` data structures - **v0.4.1** 2023-04-13 - Added `decrypt` function and set of test cases - `hash` function now returns `null` instead of `-1` when an invalid hashVersion is specified - Fixed broken feature test case (was using `[0.99]` instead of `0.99` for coverage) - **v0.4.2** 2023-04-30 - Add test cases when targeting condition value is `null` - **v0.5.0** 2023-05-17 - Add support for new version string comparison operators (`$veq`, `$vne`, `$vgt`, `$vgte`, `$vlt`, `$vlte`) and new `paddedVersionString` helper function - New `isIn` helper function for conditions, plus new evalCondition test cases for `$in` and `$nin` operators when attribute is an array - **v0.5.1** 2023-10-19 - Add 2 new test cases for matching on a `$groups` array attribute - **v0.5.2** 2023-10-30 - Add 3 new test cases for comparison operators to handle more edge cases - **v0.5.3** 2024-01-02 - Experiment conditions are now evaluated within the experiment object - New `fallbackAttribute`, `disableStickyBucketing`, `bucketVersion`, `minBucketVersion`, properties of FeatureRules - New `fallbackAttribute`, `disableStickyBucketing`, `bucketVersion`, `minBucketVersion`, properties of Experiments - Add `stickyBucketUsed` to ExperimentResult object - **v0.5.4** 2024-02-23 - New `parentConditions` property of FeatureRules - New `parentConditions` property of Experiments - **v0.5.5** 2024-04-09 - Add test cases for URL Redirects - Add `navigate` method to Context - Add `persistQueryString` property to Experiment - Add and improve test cases for StickyBucket - **v0.6.0** 2024-04-30 - Remove `versionCompare` test cases (these are now just included as part of `evalCondition`) - Tweak to `isIncludedInRollout` to handle an edge case when coverage is zero. Also added test case for this. - Add `id` property to feature rules (reserved for future use) - **v0.6.1** 2024-05-13 - Update logic in `evalCondition` to allow for and/or/not/nor operators to appear at the same level as other conditions - Added test cases for multiple operators on the same level - **v0.7.0** 2024-06-25 - New Operators `$inGroup` and `$notInGroup` to check Saved Groups by reference - Add argument to `evalCondition` for definition of Saved Groups - Add test cases for `evalCondition`, `feature`, and `run` using the new operators - **v0.7.1** 2025-02-13 - Add feature rule id to `FeatureResult` object and updated all test cases to include it - New tests for pre-requisite edge cases - New tests to make sure only known properties are copied from feature rules to experiments - New tests for evalCondition to cover null/false edge cases --- ## C# SDK import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # C# The GrowthBook C# SDK supports all modern .NET platforms including .NET 6+, .NET Framework 4.6.1+, and .NET Standard 2.0+. ## Installation Install via NuGet Package Manager: ```bash dotnet add package growthbook-c-sharp ``` Or via Package Manager Console: ```powershell Install-Package growthbook-c-sharp ``` ## Quick Start Get started with GrowthBook in just a few steps: ```csharp using GrowthBook; using Newtonsoft.Json.Linq; // 1. Create a context with user attributes var context = new Context { Enabled = true, Attributes = new JObject { ["id"] = "user-123", ["country"] = "US", ["plan"] = "premium" } }; // 2. Initialize GrowthBook var gb = new GrowthBook.GrowthBook(context); // 3. Load features from API (async) await gb.LoadFeaturesAsync("https://cdn.growthbook.io", "sdk_abc123"); // 4. Evaluate features if (gb.IsOn("new-dashboard")) { ShowNewDashboard(); } var buttonColor = gb.GetFeatureValue("button-color", "blue"); var maxRetries = gb.GetFeatureValue("max-retries", 3); ``` ## Loading Features from API ### Basic API Integration ```csharp using System.Net.Http; using Newtonsoft.Json; public class FeaturesResult { public HttpStatusCode Status { get; set; } public IDictionary? Features { get; set; } public DateTimeOffset? DateUpdated { get; set; } } public async Task InitializeGrowthBookAsync() { var context = new Context { Enabled = true, Attributes = GetUserAttributes() }; var gb = new GrowthBook.GrowthBook(context); // Load features from API using var httpClient = new HttpClient(); var url = "https://cdn.growthbook.io/api/features/sdk_abc123"; var response = await httpClient.GetAsync(url); if (response.IsSuccessStatusCode) { var content = await response.Content.ReadAsStringAsync(); var featuresResult = JsonConvert.DeserializeObject(content); // Update context with loaded features context.Features = featuresResult.Features; gb.UpdateContext(context); } return gb; } private JObject GetUserAttributes() { return new JObject { ["id"] = User.Identity.Name, ["email"] = User.Email, ["country"] = User.Country, ["plan"] = User.SubscriptionPlan }; } ``` ### Streaming Updates Enable real-time feature updates with Server-Sent Events (SSE): ```csharp using System.Threading; using System.Threading.Tasks; public class GrowthBookManager : IDisposable { private readonly GrowthBook.GrowthBook _gb; private readonly Timer _refreshTimer; private readonly HttpClient _httpClient; public GrowthBookManager(Context context) { _gb = new GrowthBook.GrowthBook(context); _httpClient = new HttpClient(); // Poll for updates every 60 seconds _refreshTimer = new Timer( async _ => await RefreshFeaturesAsync(), null, TimeSpan.Zero, TimeSpan.FromSeconds(60) ); } private async Task RefreshFeaturesAsync() { try { var url = "https://cdn.growthbook.io/api/features/sdk_abc123"; var response = await _httpClient.GetAsync(url); if (response.IsSuccessStatusCode) { var content = await response.Content.ReadAsStringAsync(); var result = JsonConvert.DeserializeObject(content); // Update features var context = _gb.GetContext(); context.Features = result.Features; _gb.UpdateContext(context); Console.WriteLine($"Features updated: {result.Features.Count} features loaded"); } } catch (Exception ex) { Console.WriteLine($"Failed to refresh features: {ex.Message}"); } } public GrowthBook.GrowthBook GetGrowthBook() => _gb; public void Dispose() { _refreshTimer?.Dispose(); _httpClient?.Dispose(); } } ``` ## User Attributes Attributes are used for targeting and experiment assignment: ```csharp // Standard attributes var attributes = new JObject { ["id"] = "user-123", ["email"] = "user@example.com", ["country"] = "US", ["browser"] = "chrome" }; // Custom business attributes var attributes = new JObject { ["id"] = user.Id, ["subscriptionTier"] = user.Tier, ["lifetimeValue"] = user.LifetimeValue, ["accountAge"] = (DateTime.Now - user.CreatedAt).Days, ["isHighValueCustomer"] = user.LifetimeValue > 1000, ["purchasedCategories"] = new JArray(user.Categories), ["enabledFeatures"] = new JArray(user.Features) }; var context = new Context { Enabled = true, Attributes = attributes }; ``` ## Evaluating Features ### Feature Result Properties ```csharp var result = gb.EvalFeature("my-feature"); // Check if feature is enabled if (result.On) { ShowNewFeature(); } // Get feature value var value = result.Value; // JToken var typedValue = result.GetValue(); // Typed accessor // Check the source switch (result.Source) { case FeatureResult.SourceId.DefaultValue: // Using default value break; case FeatureResult.SourceId.Force: // Forced value break; case FeatureResult.SourceId.Experiment: // Value from experiment var experiment = result.Experiment; var experimentResult = result.ExperimentResult; TrackExperiment(experiment, experimentResult); break; } ``` ### Generic Type Accessors The SDK provides type-safe generic methods: ```csharp // Get feature values with type safety var isEnabled = gb.GetFeatureValue("new-feature", false); var buttonColor = gb.GetFeatureValue("button-color", "blue"); var maxRetries = gb.GetFeatureValue("max-retries", 3); var timeout = gb.GetFeatureValue("api-timeout", 5.0); // Complex types var config = gb.GetFeatureValue>("app-config", null); if (config != null) { var apiKey = config["apiKey"]?.ToString(); var maxConnections = Convert.ToInt32(config["maxConnections"]); } ``` ## Running Experiments ### Inline Experiments ```csharp var experiment = new Experiment { Key = "button-color-test", Variations = new JArray { "blue", "red", "green" }, Weights = new List { 0.5, 0.3, 0.2 } }; var result = gb.Run(experiment); if (result.InExperiment) { var color = result.GetValue(); SetButtonColor(color); // Track experiment view TrackExperiment(experiment, result); } ``` ### Experiment Configuration ```csharp var experiment = new Experiment { // Required Key = "pricing-test", Variations = new JArray { 9.99, 14.99, 19.99 }, // Optional configuration Active = true, Coverage = 0.8, // 80% of users Weights = new List { 0.5, 0.3, 0.2 }, // Targeting Condition = JObject.Parse(@"{""country"": ""US"", ""plan"": ""premium""}"), HashAttribute = "id", // Sticky bucketing BucketVersion = 1, MinBucketVersion = 0, DisableStickyBucketing = false }; var result = gb.Run(experiment); ``` ## Encryption & Security ### Encrypted Features Enable encryption for sensitive feature configurations: ```csharp // Load encrypted features var decryptionKey = Environment.GetEnvironmentVariable("GROWTHBOOK_DECRYPTION_KEY"); await gb.LoadFeaturesAsync( apiHost: "https://cdn.growthbook.io", clientKey: "sdk_abc123", httpClient: new HttpClient(), decryptionKey: decryptionKey ); ``` ### Secure Attributes Hash sensitive attributes before sending them to GrowthBook: ```csharp using System.Security.Cryptography; using System.Text; public class SecureAttributeHelper { private readonly string _salt; public SecureAttributeHelper(string salt) { _salt = salt; } public string HashAttribute(string value) { using var sha256 = SHA256.Create(); var bytes = Encoding.UTF8.GetBytes(value + _salt); var hash = sha256.ComputeHash(bytes); return BitConverter.ToString(hash).Replace("-", "").ToLowerInvariant(); } public JObject BuildSecureAttributes(User user) { return new JObject { ["id"] = user.Id, // Hash sensitive attributes ["email"] = HashAttribute(user.Email), ["phone"] = HashAttribute(user.Phone), // Non-sensitive attributes remain plain ["country"] = user.Country, ["plan"] = user.Plan }; } } // Usage var helper = new SecureAttributeHelper( Environment.GetEnvironmentVariable("GROWTHBOOK_SECURE_ATTRIBUTE_SALT") ); var context = new Context { Enabled = true, Attributes = helper.BuildSecureAttributes(currentUser) }; ``` ### Security Best Practices ```csharp // Store keys securely in configuration public class GrowthBookConfiguration { public string ClientKey { get; set; } public string DecryptionKey { get; set; } public string SecureAttributeSalt { get; set; } } // In Startup.cs or Program.cs services.Configure( Configuration.GetSection("GrowthBook") ); // Use in services public class GrowthBookService { private readonly GrowthBookConfiguration _config; public GrowthBookService(IOptions config) { _config = config.Value; } public async Task CreateGrowthBookAsync() { var context = new Context { Enabled = true }; var gb = new GrowthBook.GrowthBook(context); await gb.LoadFeaturesAsync( "https://cdn.growthbook.io", _config.ClientKey, decryptionKey: _config.DecryptionKey ); return gb; } } ``` **Security Recommendations:** - **Never hardcode** encryption keys or salts in source code - **Use configuration** providers (appsettings.json, environment variables, Azure Key Vault) - **Rotate keys regularly** and coordinate updates across all environments - **Use different keys** for each environment (dev, staging, production) ## Sticky Bucketing Sticky bucketing ensures consistent experiment variations across sessions: ```csharp public interface IStickyBucketService { Task GetAssignmentsAsync(string attributeName, string attributeValue); Task SaveAssignmentsAsync(StickyBucketAssignmentDoc doc); Task> GetAllAssignmentsAsync(Dictionary attributes); } public class StickyBucketAssignmentDoc { public string AttributeName { get; set; } public string AttributeValue { get; set; } public Dictionary Assignments { get; set; } } ``` ### Implementation Example ```csharp using Microsoft.Extensions.Caching.Memory; public class MemoryStickyBucketService : IStickyBucketService { private readonly IMemoryCache _cache; private readonly string _prefix = "gb_sticky_"; public MemoryStickyBucketService(IMemoryCache cache) { _cache = cache; } public Task GetAssignmentsAsync( string attributeName, string attributeValue) { var key = $"{_prefix}{attributeName}||{attributeValue}"; _cache.TryGetValue(key, out StickyBucketAssignmentDoc doc); return Task.FromResult(doc); } public Task SaveAssignmentsAsync(StickyBucketAssignmentDoc doc) { var key = $"{_prefix}{doc.AttributeName}||{doc.AttributeValue}"; _cache.Set(key, doc, TimeSpan.FromDays(30)); return Task.CompletedTask; } public Task> GetAllAssignmentsAsync( Dictionary attributes) { var docs = new Dictionary(); foreach (var (attrName, attrValue) in attributes) { var doc = GetAssignmentsAsync(attrName, attrValue).Result; if (doc != null) { var docKey = $"{doc.AttributeName}||{doc.AttributeValue}"; docs[docKey] = doc; } } return Task.FromResult(docs); } } // Configure in Startup.cs services.AddMemoryCache(); services.AddSingleton(); // Use with GrowthBook var context = new Context { Enabled = true, StickyBucketService = stickyBucketService, Attributes = attributes }; ``` ## Remote Evaluation :::tip See the [Remote Evaluation overview](/self-host/remote-evaluation) for more information about what Remote Evaluation is, how it works, and deployment options. ::: Remote evaluation evaluates feature flags on a secure server: ```csharp public class RemoteEvaluationService { private readonly HttpClient _httpClient; private readonly string _apiHost; private readonly string _clientKey; public RemoteEvaluationService(HttpClient httpClient, string apiHost, string clientKey) { _httpClient = httpClient; _apiHost = apiHost; _clientKey = clientKey; } public async Task> EvaluateFeaturesAsync(JObject attributes) { var url = $"{_apiHost}/api/eval/{_clientKey}"; var payload = new { attributes = attributes }; var response = await _httpClient.PostAsJsonAsync(url, payload); response.EnsureSuccessStatusCode(); var result = await response.Content.ReadAsAsync>(); return result; } } ``` ## Async API Support The C# SDK now provides comprehensive async APIs for non-blocking operations: ### Async Feature Loading ```csharp // Load features asynchronously await gb.LoadFeaturesAsync(apiHost, clientKey); // Load features with custom HTTP client using var httpClient = new HttpClient(); await gb.LoadFeaturesAsync(apiHost, clientKey, httpClient); // Load encrypted features await gb.LoadFeaturesAsync(apiHost, clientKey, httpClient, decryptionKey: "key_abc123"); ``` ### Async Feature Refresh ```csharp // Refresh features in the background await gb.RefreshFeaturesAsync(); // Refresh with custom timeout using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)); await gb.RefreshFeaturesAsync(cts.Token); ``` ### Task-Based Patterns All async operations return `Task` or `Task` for seamless integration with async/await: ```csharp public async Task Index() { var gb = GetGrowthBookInstance(); // Non-blocking feature evaluation var features = await Task.Run(() => new { NewDashboard = gb.IsOn("new-dashboard"), MaxItems = gb.GetFeatureValue("max-items", 10), Theme = gb.GetFeatureValue("theme", "light") }); return View(features); } ``` ## Experiment Tracking Implement tracking callbacks to send experiment data to your analytics: ```csharp public class GrowthBookWithTracking { private readonly GrowthBook.GrowthBook _gb; private readonly IAnalyticsService _analytics; public GrowthBookWithTracking(Context context, IAnalyticsService analytics) { _gb = new GrowthBook.GrowthBook(context); _analytics = analytics; // Subscribe to tracking events context.TrackingCallback = TrackExperiment; } private void TrackExperiment(Experiment experiment, ExperimentResult result) { if (result.InExperiment) { _analytics.Track("experiment_viewed", new { experiment_id = experiment.Key, variation_id = result.VariationId, variation_value = result.Value, user_id = result.HashValue }); } } public GrowthBook.GrowthBook GetGrowthBook() => _gb; } ``` ### Tracking with Experiments from Features ```csharp var featureResult = gb.EvalFeature("premium-feature"); if (featureResult.Source == FeatureResult.SourceId.Experiment) { var experiment = featureResult.Experiment; var result = featureResult.ExperimentResult; // Track to analytics analytics.Track("experiment_viewed", new { experiment_id = experiment.Key, variation_id = result.VariationId, feature_id = result.FeatureId, user_id = result.HashValue, in_experiment = result.InExperiment, hash_used = result.HashUsed }); } ``` ## Troubleshooting & Logging ### Diagnostic Logging ```csharp using Microsoft.Extensions.Logging; public class GrowthBookLogger { private readonly ILogger _logger; private readonly GrowthBook.GrowthBook _gb; public GrowthBookLogger(GrowthBook.GrowthBook gb, ILogger logger) { _gb = gb; _logger = logger; } public void LogContext() { var context = _gb.GetContext(); _logger.LogInformation( "GrowthBook Context: Enabled={Enabled}, Features={FeatureCount}, URL={Url}", context.Enabled, context.Features?.Count ?? 0, context.Url ); } public void LogFeatureEvaluation(string featureKey, FeatureResult result) { _logger.LogDebug( "Feature {FeatureKey}: On={On}, Source={Source}, Value={Value}", featureKey, result.On, result.Source, result.Value ); } public void LogExperiment(Experiment experiment, ExperimentResult result) { _logger.LogInformation( "Experiment {ExperimentKey}: InExperiment={InExperiment}, VariationId={VariationId}, Value={Value}", experiment.Key, result.InExperiment, result.VariationId, result.Value ); } } ``` ### Common Issues #### Features Not Loading ```csharp public async Task VerifyFeaturesLoadedAsync() { try { var context = _gb.GetContext(); if (context.Features == null || context.Features.Count == 0) { _logger.LogWarning("No features loaded in GrowthBook context"); return false; } _logger.LogInformation("Features loaded: {Count}", context.Features.Count); return true; } catch (Exception ex) { _logger.LogError(ex, "Error verifying features"); return false; } } ``` #### Decryption Failures ```csharp public async Task TestDecryptionAsync() { try { var decryptionKey = Environment.GetEnvironmentVariable("GROWTHBOOK_DECRYPTION_KEY"); if (string.IsNullOrEmpty(decryptionKey)) { _logger.LogError("GROWTHBOOK_DECRYPTION_KEY not set"); return false; } await _gb.LoadFeaturesAsync( "https://cdn.growthbook.io", "sdk_abc123", decryptionKey: decryptionKey ); _logger.LogInformation("Successfully loaded encrypted features"); return true; } catch (Exception ex) { _logger.LogError(ex, "Failed to load encrypted features"); return false; } } ``` ### Health Checks ```csharp using Microsoft.Extensions.Diagnostics.HealthChecks; public class GrowthBookHealthCheck : IHealthCheck { private readonly GrowthBook.GrowthBook _gb; public GrowthBookHealthCheck(GrowthBook.GrowthBook gb) { _gb = gb; } public Task CheckHealthAsync( HealthCheckContext context, CancellationToken cancellationToken = default) { try { var gbContext = _gb.GetContext(); if (!gbContext.Enabled) { return Task.FromResult( HealthCheckResult.Degraded("GrowthBook is disabled") ); } var featureCount = gbContext.Features?.Count ?? 0; if (featureCount == 0) { return Task.FromResult( HealthCheckResult.Degraded("No features loaded") ); } return Task.FromResult( HealthCheckResult.Healthy($"{featureCount} features loaded") ); } catch (Exception ex) { return Task.FromResult( HealthCheckResult.Unhealthy("GrowthBook health check failed", ex) ); } } } // Register in Startup.cs services.AddHealthChecks() .AddCheck("growthbook"); ``` ## Integrations ### ASP.NET Core Integrate GrowthBook with ASP.NET Core for feature flags in your web application: ```csharp // Startup.cs or Program.cs public class Startup { public void ConfigureServices(IServiceCollection services) { // Register GrowthBook as singleton services.AddSingleton(sp => { var context = new Context { Enabled = true, Attributes = new JObject() }; return new GrowthBook.GrowthBook(context); }); // Register background service for feature refresh services.AddHostedService(); services.AddControllersWithViews(); } } // Background service for periodic refresh public class GrowthBookRefreshService : BackgroundService { private readonly GrowthBook.GrowthBook _gb; private readonly ILogger _logger; public GrowthBookRefreshService( GrowthBook.GrowthBook gb, ILogger logger) { _gb = gb; _logger = logger; } protected override async Task ExecuteAsync(CancellationToken stoppingToken) { // Initial load await RefreshFeaturesAsync(); while (!stoppingToken.IsCancellationRequested) { await Task.Delay(TimeSpan.FromSeconds(60), stoppingToken); await RefreshFeaturesAsync(); } } private async Task RefreshFeaturesAsync() { try { using var httpClient = new HttpClient(); var url = "https://cdn.growthbook.io/api/features/sdk_abc123"; var response = await httpClient.GetAsync(url); if (response.IsSuccessStatusCode) { var content = await response.Content.ReadAsStringAsync(); var result = JsonConvert.DeserializeObject(content); var context = _gb.GetContext(); context.Features = result.Features; _gb.UpdateContext(context); _logger.LogInformation("Features refreshed: {Count}", result.Features.Count); } } catch (Exception ex) { _logger.LogError(ex, "Failed to refresh features"); } } } // Middleware for adding user context public class GrowthBookMiddleware { private readonly RequestDelegate _next; public GrowthBookMiddleware(RequestDelegate next) { _next = next; } public async Task InvokeAsync(HttpContext context, GrowthBook.GrowthBook gb) { // Build user attributes from HTTP context var attributes = new JObject { ["id"] = context.User.Identity?.Name, ["country"] = context.Request.Headers["CF-IPCountry"].FirstOrDefault(), ["userAgent"] = context.Request.Headers["User-Agent"].FirstOrDefault(), ["url"] = context.Request.Path.Value }; // Update GrowthBook context for this request var gbContext = gb.GetContext(); gbContext.Attributes = attributes; gb.UpdateContext(gbContext); // Store in HttpContext for controller access context.Items["GrowthBook"] = gb; await _next(context); } } // Use in controller public class HomeController : Controller { private readonly GrowthBook.GrowthBook _gb; public HomeController(GrowthBook.GrowthBook gb) { _gb = gb; } public IActionResult Index() { // Use feature flags var showNewDashboard = _gb.IsOn("new-dashboard"); var maxItems = _gb.GetFeatureValue("dashboard-max-items", 10); // Track experiment var colorResult = _gb.EvalFeature("dashboard-theme-color"); if (colorResult.Source == FeatureResult.SourceId.Experiment) { TrackExperiment(colorResult.Experiment, colorResult.ExperimentResult); } return View(new DashboardViewModel { ShowNewDashboard = showNewDashboard, MaxItems = maxItems, ThemeColor = colorResult.GetValue() }); } private void TrackExperiment(Experiment experiment, ExperimentResult result) { // Track to your analytics service Analytics.Track(User.Identity.Name, "experiment_viewed", new { experiment_id = experiment.Key, variation_id = result.VariationId }); } } ``` ### Blazor Server Use GrowthBook with Blazor Server for reactive feature flags: ```csharp // Program.cs builder.Services.AddSingleton(); builder.Services.AddScoped(); // GrowthBookService.cs public class GrowthBookService { private readonly GrowthBook.GrowthBook _gb; private readonly ILogger _logger; public GrowthBookService(ILogger logger) { _logger = logger; var context = new Context { Enabled = true }; _gb = new GrowthBook.GrowthBook(context); // Start background refresh _ = RefreshFeaturesAsync(); } public GrowthBook.GrowthBook GetGrowthBook() => _gb; public async Task RefreshFeaturesAsync() { try { using var httpClient = new HttpClient(); await _gb.LoadFeaturesAsync( "https://cdn.growthbook.io", "sdk_abc123", httpClient ); _logger.LogInformation("Features loaded"); } catch (Exception ex) { _logger.LogError(ex, "Failed to load features"); } } public event EventHandler FeaturesUpdated; protected virtual void OnFeaturesUpdated() { FeaturesUpdated?.Invoke(this, EventArgs.Empty); } } // Blazor component @page "/dashboard" @inject GrowthBookService GrowthBookService @implements IDisposable Dashboard @if (_newDashboard) { } else { } @code { private bool _newDashboard; private int _maxItems; protected override void OnInitialized() { UpdateFeatures(); GrowthBookService.FeaturesUpdated += OnFeaturesUpdated; } private void OnFeaturesUpdated(object sender, EventArgs e) { UpdateFeatures(); StateHasChanged(); } private void UpdateFeatures() { var gb = GrowthBookService.GetGrowthBook(); _newDashboard = gb.IsOn("new-dashboard"); _maxItems = gb.GetFeatureValue("max-items", 20); } public void Dispose() { GrowthBookService.FeaturesUpdated -= OnFeaturesUpdated; } } ``` ## Supported Features --- ## Cloudflare Workers Edge App & SDK import ExternalLink from '@site/src/components/ExternalLink'; import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Cloudflare Workers Edge App & SDK ## Overview GrowthBook currently supports two levels of integration with most edge workers, including Cloudflare: 1. Our turnkey Edge App - Automatically run server-side or hybrid [Visual Experiments](/app/visual) without redraw flicker. - Automatically run server-side or hybrid [URL Redirect Experiments](/app/url-redirects) without flicker or delay. - Perform custom feature flagging and experimentation logic. - Optionally inject the JavaScript SDK with hydrated payload, allowing the front-end to pick up where the edge left off without any extra network requests. We use an enhanced version of our [HTML Script Tag](/lib/script-tag) for this purpose. 2. Support for edge apps using our JavaScript SDK - Enhanced support and examples for using our JavaScript SDK in an edge environment Regardless of your use case, our Cloudflare integration makes easy to synchronize feature and experiment values between GrowthBook and Cloudflare's KV store. This eliminates the network request to the GrowthBook API, unlocking blazingly fast edge-side and client-side SDK performance. ## References - Our Cloudflare Workers SDK repository, which supports the above use cases, is [here ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-cloudflare) - A turnkey implementation of the Edge App (compatible with Wrangler) is [here ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-cloudflare/example) - You may find it useful to review our [JavaScript SDK](/lib/js). Many of the concepts which apply to both on-edge and injected frontend SDKs are based on our JS SDK. ## Worker Configuration :::tip Get up to speed with Cloudflare Workers This tutorial assumes some familiarity with building and deploying Cloudflare Worker applications. You can quickly get up to speed by following the Cloudflare Workers [Getting Started ](https://developers.cloudflare.com/workers/get-started/guide/) guide. ::: You may either use our turnkey Edge App for Cloudflare Workers or build your own app from scratch using our JavaScript and Cloudflare SDKs. ## Turnkey Edge App Our Edge App runs as a smart proxy layer between your application and your end users. In absence of Visual or URL Redirect experiments, the Edge App will simply proxy the user request to your site and return the response, optionally injecting a fully-bootstrapped JavaScript SDK onto the rendered HTML page. If the request URL matches an Visual or URL Redirect experiment and the targeting conditions are satisfied, the Edge App may also perform one or more URL redirects behind the scenes (the public-facing URL does not change) and/or mutate the DOM for Visual Experiments. Additionally, by using [lifecycle hooks](#lifecycle-hooks) you can perform custom logic such as feature flagging as well as proxying and early returns. :::note URL Redirects on edge The Edge App defaults to running URL Redirect Experiments in the browser only. This is because edge redirects load a separate page's content without altering the URL. After the redirect, some sites may experience problems with loading assets or endpoints with relative paths. You can enable URL Redirects on edge by setting environment variable `RUN_URL_REDIRECT_EXPERIMENTS` to "edge" or "everywhere". Additionally if your redirect is cross-domain (e.g. redirection from "public.mysite.io" to "newsite.io"), you must also set `RUN_CROSS_ORIGIN_URL_REDIRECT_EXPERIMENTS`. See [environment variables](#environment-variables) for more information. ::: Setting up our turnkey Edge App is simple. Assuming that you have a basic Worker application set up, simply install the SDK and implement our custom request handler. Or if you prefer, you may pull down our fully-functional [example implementation ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-cloudflare/example) and follow along. ### Install the SDK ```bash npm2yarn npm install --save @growthbook/edge-cloudflare ``` ### Implement the Edge App request handler A basic implementation of our Edge App only requires a few lines of code: ```ts import { handleRequest } from "@growthbook/edge-cloudflare"; export default { fetch: async function (request, env, ctx) { return await handleRequest(request, env); }, }; ``` ### Configure the Edge App Use a combination of environment variables and optional runtime configuration to add required fields and to customize the Edge App behavior. #### Environment variables Edit your `wrangler.toml` file and, at minimum, add these required fields: ```bash [vars] PROXY_TARGET="https://internal.mysite.io" # The non-edge URL to your website GROWTHBOOK_API_HOST="https://cdn.growthbook.io" GROWTHBOOK_CLIENT_KEY="sdk-abc123" GROWTHBOOK_DECRYPTION_KEY="key_abc123" # Only include for encrypted SDK Connections ``` You may want to further customize the app. Here is a list of common customization variables: ```bash # Disable or change the rendering behavior of Visual Experiments: # ========== RUN_VISUAL_EDITOR_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "everywhere" # URL Redirect Experiments are disabled on edge by default. Because the URL does not change, some sites # may experience problems with loading assets or endpoints with relative paths: # ========== RUN_URL_REDIRECT_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "browser" RUN_CROSS_ORIGIN_URL_REDIRECT_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "browser" # Mutate browser URL via window.history.replaceState() to reflect the new URL: INJECT_REDIRECT_URL_SCRIPT="true" # default "true". # Do not inject a bootstrapped JavaScript SDK onto the page: # ========== DISABLE_INJECTIONS="true" # default "false" # Customize the edge or injected browser SDK behavior: # ========== ENABLE_STREAMING="true" # default "false". Streaming SSE updates on browser. ENABLE_STICKY_BUCKETING="true" # default "false". Use cookie-based sticky bucketing on edge and browser. ``` #### Runtime configuration You may want to provide context to your edge app at runtime rather than using environment variables. For example, if you have additional [targeting attributes](/lib/js#attributes) available, you may inject them by modifying your request handler code: ```ts import { handleRequest } from "@growthbook/edge-cloudflare"; import { parse } from "cookie"; export default { fetch: async function (request, env, ctx) { const cookie = parse(request.headers.get("Cookie") || ""); const config = { attributes: { userType: cookie["userId"] ? "logged in" : "anonymous" } }; return await handleRequest(request, env, config); }, }; ``` #### More customization options For a full list of customizations, view our vendor-agnostic [Edge Utility repository ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-utils). ### Set up a Payload Cache You can configure GrowthBook payload caching by using a [Cloudflare KV ](https://developers.cloudflare.com/kv/reference/how-kv-works/) store. This eliminates network requests from your edge to GrowthBook which speeds up page delivery while reducing network costs. Our Cloudflare Edge App will automatically use either webhook-based or just-in-time payload caching (or both) depending on how you've set up your KV namespaces, bindings, and SDK Webhooks. More information about setting up your payload cache can be found in the [Payload Caching with Cloudflare KV Store](#payload-caching-with-cloudflare-kv-store) doc section below. ### Tracking Experiment Views Running A/B tests requires a [tracking callback](/lib/js#experimentation-ab-testing). Our turnkey Edge App defaults to using built-in front-end tracking. The tracking call automatically integrates with Segment.io, GA4, and Google Tag Manager by using the mechanism outlined in our [HTML Script Tag](/lib/script-tag#tracking-experiment-views). In order to do this, the app keeps track of tracking calls triggered on edge and injects them into the front-end SDK to be automatically triggered on page load. You may wish to either customize front-end tracking or switch to edge tracking (or use both concurrently if running hybrid edge + front-end experiments). Why might you be interested in tracking on edge? Tracking on an edge or backend environment allows you to ensure the callback is fired before any differentiation across variations, eliminating experimental bias. While not eliminating this risk, the default injected front-end tracking introduced by our Edge App does reduce this risk relative to solely using a front-end SDK. To change the front-end tracking callback, set the `GROWTHBOOK_TRACKING_CALLBACK` to your custom tracking JS code: ```bash # todo: replace with your own tracking library GROWTHBOOK_TRACKING_CALLBACK="(experiment, results) => { console.log('browser tracking callback', {experiment, results}); }" ``` To track on edge, you must inject your own tracking callback into the edge request handler code. Any experiments that run on edge will use the edge tracking callback and not the front-end callback (hybrid edge + front-end experiments being an exception): ```ts import { handleRequest } from "@growthbook/edge-cloudflare"; export default { fetch: async function (request, env, ctx) { const config = { edgeTrackingCallback: (experiment, results) => { // todo: replace with your tracking library console.log('edge tracking callback', {experiment, results}); } }; return await handleRequest(request, env, config); }, }; ``` ### Targeting Attributes The following targeting attributes are set automatically by the Edge App. - `id` - a long-lived user identifier, stored in a `gbuuid` cookie (see [Cookie Policy and GDPR](#cookie-policy-and-gdpr) for how the cookie is written and when to use `PERSIST_UUID`) - `url` - `path` - `host` - `query` - `pageTitle` - `deviceType` - either `mobile` or `desktop` - `browser` - one of `chrome`, `edge`, `firefox`, `safari`, or `unknown` - `utmSource` - `utmMedium` - `utmCampaign` - `utmTerm` - `utmContent` You can customize both the primary identifier name (`id`) and cookie name (`gbuuid`) by setting the `UUID_KEY` and `UUID_COOKIE_NAME` environment variables respectively. As shown in the [runtime configuration](#runtime-configuration) section above, you can also pass custom attributes via runtime config. You can also skip automatic attribute generation and rely solely on custom attributes by setting the environment variable `SKIP_AUTO_ATTRIBUTES="true"`. ### Routing By default, the Edge App will process all `GET` requests (other HTTP verbs are proxied through without running through our app logic). It is generally preferable to configure your routing rules outside of our Edge App. For instance, you may only want to invoke the Edge App at `https://yourdomain.io/landing-page`. You can configure Cloudflare routing by following the Cloudflare Workers [Routes ](https://developers.cloudflare.com/workers/configuration/routing/routes/) documentation. There may be situations when you will need to provide finer-grained routing / URL targeting rules within our Edge App. You will need to include a JSON encoded string of route rules in your `ROUTES` environment variable. For instance, you may want to do a proxy pass-through (do not process) for `mysite.io/account/*` or `mysite.io/settings/*`. Your routes may look like this: ```bash ROUTES='[{ "pattern":"mysite.io/account/*", "behavior":"proxy" }, { "pattern":"mysite.io/settings/*", "behavior":"proxy" }]' ``` A route uses the following interface, with many of the properties being optional: ```ts { pattern: string; type?: "regex" | "simple"; // default: "simple" behavior?: "intercept" | "proxy" | "error"; // default: "intercept" includeFileExtensions?: boolean; // Include requests to filenames like "*.jpg". default: false (pass-through). statusCode?: number; // Alter the status code (default is 404 when using "error") body?: string; // Alter the body (for setting an error message body) } ``` When multiple routes are included in your `ROUTES` array, only the first match is used. ### Cookie Policy and GDPR The Edge App uses a random unique identifier, stored in a first-party cookie named `gbuuid`, to provide a consistent user experience to your visitors by preventing them from being re-bucketed into different A/B test variations. It follows the same mechanism as discussed in our [HTML Script Tag docs](/lib/script-tag#cookie-policy-and-gdpr). #### Persisting the `gbuuid` Cookie from the Edge By default, the Edge App generates a `gbuuid` for each visitor and exposes it as the `id` targeting attribute, but it does **not** write the cookie from the edge server itself. Instead, the cookie is persisted in the browser by the injected front-end SDK once the page loads. This works for most setups. However, the `gbuuid` cookie will **not** be written if any of the following are true: - You have disabled SDK injection (`DISABLE_INJECTIONS="true"`). - You need the `gbuuid` cookie to be available on the very first request — for example, before the browser has a chance to execute the injected SDK. In these cases, set the environment variable `PERSIST_UUID="true"` (via `[vars]` in your `wrangler.toml`). The Edge App will then include a `Set-Cookie: gbuuid=...` header on its response, ensuring the cookie is stored on the first request. If you are using `NO_AUTO_COOKIES` to wait for consent (see the [next section](#delay-storing-the-cookie-until-consent-is-granted)), no cookies — including `gbuuid` — will be written until the `growthbookpersist` event is dispatched, regardless of `PERSIST_UUID`. #### Delay Storing the Cookie Until Consent is Granted If you must delay persisting the `gbuuid` cookie until a user consents, you can set the environment variable `NO_AUTO_COOKIES="true"`. This will still generate a UUID for the user, but will not persist it. That means, if the user refreshes the page, they will have a new random UUID generated.environment You have the option to manually persist this cookie at any time, for example when a user grants consent on your cookie banner. All you need to do is fire this custom event from javascript on the rendered page: ```js document.dispatchEvent(new CustomEvent("growthbookpersist")); ``` :::note Sticky Bucketing and consent If you are using Sticky Bucketing, a persistent sticky bucket assignments cookie will automatically be generated. If you require user permission before writing cookies, you should: - Either do not enable Sticky Bucketing on edge (do not use `ENABLE_STICKY_BUCKETING`) - Or only enable Sticky Bucketing per each user via runtime configuration. (only pass `config.enableStickyBucketing: true` if user has consented — identifiable by checking for presence of the `gbuuid` cookie). ::: ### Lifecycle hooks You can perform custom logic and optionally return a response at various stages in the Edge App's lifecycle. This allows for expressiveness of custom routing, user attribute mutation, header and body (DOM) mutation, and custom feature flag and experiment implementations – while preserving the ability to automatically run Visual and URL Redirect experiments and SDK hydration. With each hook, you may mutate any of the provided attributes _or_ return an early response to halt the Edge App processing. The following hooks are available: - `onRequest` - Fired on initial user request. Can exit early based on requested URL. - `onRoute` - Fired after standard routing has been processed. Can exit early (proxy) based on manual routing logic. - `onUserAttributes` - Fired after auto-attributes have been assigned to the user. Either enhance the provided `attributes` object or exit early if desired. - `onGrowthBookInit` - Fired after the Edge App's internal GrowthBook SDK has been initialized. Call SDK functions or exit early if desired. - `onBeforeOriginFetch` - Similar hook to the above; triggers after any URL Redirect experiments have run but before any origin requests have been made. - `onOriginFetch` - Fired immediately after the origin fetch has been made, but before the full response body has been captured. Useful for exiting early based on response status or headers. - `onBodyReadyParams` - Fired once the entire response body has been parsed. In addition to early exiting, you may begin to mutate the final response body via `resHeaders` and the `setBody()` method. The text `body` as well as the optional parsed virtual DOM `root` (disabled by default, use `ALWAYS_PARSE_DOM` to enable) are exposed. NOTE: If mutating the `root` DOM, it is your responsibility to `setBody()` with the latest changes before the response is returned. - `onBeforeResponse` - The final hook fired before the response is returned to the user, triggering after both visual editor changes and client SDK hydration have been injected. While the virtual DOM is no longer available, this hook can be used to apply any final changes the body via `setBody()`. To use one or more lifecycle hooks, pass any hooks to your `handleRequest` method: ```ts const hooks = { onRoute: (params) => { if (params.requestUrl === "https://mysite.io/skip") { return params.context.helpers.proxyRequest(context, params.req, params.res); } }, onBeforeResponse: (params) => { params.setBody(params.body + `console.log("custom logic")`); } }; export default { fetch: async function (request, env, ctx) { return await handleRequest(request, env, undefined, hooks); }, }; ``` ## Manual SDK Integration on Edge You may be interested in building your own edge application using the GrowthBook SDK and not using our turnkey Edge App. Or you may want to do custom feature flagging on specific routes while running our Edge App on other routes. To use the GrowthBook on edge, simply include our standard [JavaScript SDK](/lib/js) (`@growthbook/growthbook` NPM package). In our `@growthbook/edge-cloudflare` NPM package, we export a few Cloudflare-specific utility functions to simplify SDK payload caching (we discuss payload caching strategies in the subsequent doc section). ```ts import { GrowthBook, setPolyfills } from "@growthbook/growthbook"; import { getPayloadFromKV, getKVLocalStoragePolyfill } from "@growthbook/edge-cloudflare"; export default { async fetch(request) { // 1. Init the GrowthBook SDK and choose an optional caching strategy // A. Use the KV as a managed payload store to eliminate SDK requests to the GrowthBook API entirely. // Requires setting up an SDK Webhook. const payload = await getPayloadFromKV(env); const growthbook = new GrowthBook(gbContext); await growthbook.init({ payload: payload }); // B. Or provide a KV cache layer so that the GrowthBook SDK doesn't need to make as many requests // to the GrowthBook API. No SDK Webhook needed. const localStoragePolyfill = getKVLocalStoragePolyfill(env); setPolyfills({ localStorage: localStoragePolyfill }); await growthbook.init(); // 2. Start feature flagging if (growthbook.isOn("my-feature")) { return new Response("foo"); } return new Response("bar"); } } ``` ## Payload Caching with Cloudflare KV Store By default, the Edge App will make a network request to the GrowthBook API on each user request in order to fetch the current feature and experiment values. This is a blocking call that delays page delivery. There is an in-memory short-lived cache layer on this call, but it won't always protect you. Convenient solutions this problem are realized through [Cloudflare KV ](https://developers.cloudflare.com/kv/reference/how-kv-works/), an on-edge key-val store which we can leverage for persistent payload caching. There are 2 levels of KV integration available: 1. You can either completely eliminate the blocking call to the GrowthBook API by implementing a GrowthBook-to-Cloudflare-KV push model via **SDK Webhooks**. 2. Alternatively, you can eliminate most of these network requests by using Cloudflare KV as a just-in-time payload cache. You can also use either of these strategies in your own manual SDK integration via the `getPayloadFromKV` and `getKVLocalStoragePolyfill` utility functions. ### Configuring the KV store Create a Cloudflare KV namespace for your worker to interface with. You can do this either using the Cloudflare dashboard or via Wrangler commands. By default, the GrowthBook Edge App and KV utility functions use the following KV namespaces; you only need to choose one, not both, depending on your desired level of integration: 1. KV stored payloads: `KV_GB_PAYLOAD` 2. KV payload cache: `KV_GB_CACHE` For KV stored payloads (1), we also assume a KV key of `"gb_payload"`. You will likely not need to modify this, but for manual implementations both the namespace and key can be specified in the utility functions. You must also create a KV binding so that your Cloudflare Worker can access the KV namespace. Edit your `wrangler.toml` file to add the binding: ```bash # You probably do not need both bindings: kv_namespaces = [ { binding = "KV_GB_PAYLOAD", id = "abcdefg1234567" }, { binding = "KV_GB_CACHE", id = "qwertyuiop12345" } ] [vars] ... ``` ### Configuring a SDK Webhook For KV stored payloads (1), we eliminate network requests from edge to GrowthBook by using a GrowthBook SDK Webhook to push the SDK payload to the KV store on change. 1. Create an [SDK Webhook](/app/webhooks/sdk-webhooks) on the same SDK Connection that you are using for edge integration. You do not need to worry about the receiving end of the webhook (verifying GrowthBook signatures, etc). 2. Select **Cloudflare KV** as the Webhook Type and fill out the following fields: - Account ID - Namespace ID - Key - API Token Now whenever feature and experiment values change, your Cloudflare worker will have immediate access to the latest values. You can also test the webhook by using the "Test Webhook" button on the SDK Connection page. Under the hood, the webhook is being configured with the following properties. If you need to change any of these settings for any reason, you can always edit the webhook. - **Endpoint URL** is being set to ``` https://api.cloudflare.com/client/v4/accounts/{accountId}/storage/kv/namespaces/{namespaceId}/values/{key} ``` - **Method** is being set to `PUT` - An **Authorization Header** is being added with your API Token - The **Payload format** is being set to `SDK Payload only` ## Supported Features --- ## Fastly Compute Edge App & SDK import ExternalLink from '@site/src/components/ExternalLink'; import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Fastly Compute Edge App & SDK ## Overview GrowthBook currently supports two levels of integration with most edge workers, including Fastly: 1. Our turnkey Edge App - Automatically run server-side or hybrid [Visual Experiments](/app/visual) without redraw flicker. - Automatically run server-side or hybrid [URL Redirect Experiments](/app/url-redirects) without flicker or delay. - Perform custom feature flagging and experimentation logic. - Optionally inject the JavaScript SDK with hydrated payload, allowing the front-end to pick up where the edge left off without any extra network requests. We use an enhanced version of our [HTML Script Tag](/lib/script-tag) for this purpose. 2. Support for edge apps using our JavaScript SDK - Enhanced support and examples for using our JavaScript SDK in an edge environment Regardless of your use case, our Fastly integration makes easy to synchronize feature and experiment values between GrowthBook and Fastly's KV store. This eliminates the network request to the GrowthBook API, unlocking blazingly fast edge-side and client-side SDK performance. ## References - Our Fastly Compute SDK repository, which supports the above use cases, is [here ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-fastly) - A turnkey implementation of the Edge App (compatible with Viceroy) is [here ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-fastly/example) - You may find it useful to review our [JavaScript SDK](/lib/js). Many of the concepts which apply to both on-edge and injected frontend SDKs are based on our JS SDK. ## Worker Configuration :::tip Get up to speed with Fastly Compute This tutorial assumes some familiarity with building and deploying Fastly Compute applications. You can quickly get up to speed by following the Fastly Compute [Developer guide ](https://www.fastly.com/documentation/guides/compute/). ::: You may either use our turnkey Edge App for Fastly Compute or build your own app from scratch using our JavaScript and Fastly SDKs. ## Turnkey Edge App Our Edge App runs as a smart proxy layer between your application and your end users. In absence of Visual or URL Redirect experiments, the Edge App will simply proxy the user request to your site and return the response, optionally injecting a fully-bootstrapped JavaScript SDK onto the rendered HTML page. If the request URL matches an Visual or URL Redirect experiment and the targeting conditions are satisfied, the Edge App may also perform one or more URL redirects behind the scenes (the public-facing URL does not change) and/or mutate the DOM for Visual Experiments. Additionally, by using [lifecycle hooks](#lifecycle-hooks) you can perform custom logic such as feature flagging as well as proxying and early returns. :::note URL Redirects on edge The Edge App defaults to running URL Redirect Experiments in the browser only. This is because edge redirects load a separate page's content without altering the URL. After the redirect, some sites may experience problems with loading assets or endpoints with relative paths. You can enable URL Redirects on edge by setting environment variable `RUN_URL_REDIRECT_EXPERIMENTS` to "edge" or "everywhere". Additionally if your redirect is cross-domain (e.g. redirection from "public.mysite.io" to "newsite.io"), you must also set `RUN_CROSS_ORIGIN_URL_REDIRECT_EXPERIMENTS`. See [environment variables](#environment-variables) for more information. ::: :::caution Visual Experiments and CPU By default, Fastly allows 50ms of CPU time per request. When running Visual Experiments on edge with Fastly, it is common to see CPU time exceed the allotted 50ms. If your responses exceed 50ms and you receive a **503** error, you have a few options: 1. Upgrade your Fastly Compute account to support longer CPU time. 2. Or set the environment variable `RUN_VISUAL_EDITOR_EXPERIMENTS="browser"`. Users will still receive a flicker-free experience because the bootstrapped SDK and DOM mutations are injected into the page `` and triggered immediately on page load. ::: Setting up our turnkey Edge App is simple. Assuming that you have a basic Fastly Compute service set up, simply install the SDK and implement our custom request handler. Or if you prefer, you may pull down our fully-functional [example implementation ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-fastly/example) and follow along. ### Install the SDK ```bash npm2yarn npm install --save @growthbook/edge-fastly ``` ### Implement the Edge App request handler A basic implementation of our Edge App only requires a few lines of code: ```ts /// import { ConfigStore } from "fastly:config-store"; import { gbHandleRequest, getConfigEnvFromStore } from "@growthbook/edge-fastly"; addEventListener("fetch", (event) => event.respondWith(handleRequest(event))); async function handleRequest(event) { const envVarsStore = new ConfigStore("env_vars"); const env = getConfigEnvFromStore(envVarsStore); const config = { apiHostBackend: "api_host", // Name of Fastly backend pointing to your GrowthBook API Endpoint backends: { "https://internal.mysite.io": "my_site" }, // Map of proxy origins to named Fastly backends }; return await gbHandleRequest(event.request, env, config); } ``` Notice a few references to backends. We will define these in the Fastly Dashboard in the subsequent doc section. ### Set up backends (origins) Unless your have requested open proxy behavior on your Fastly account, Fastly requires that you define backends for each origin server that your Compute application fetches from. In order to use our Edge app, you will need to create a backend for your GrowthBook API Host (which we'll call `api_host`), and one or more backends for your origin site using the Fastly dashboard (called "Origins" in your Compute service configuration). Each backend is defined as an _origin_ URL. Example: `https://internal.mysite.io` or `https://internal.mobile.mysite.io`; but not full URLs like `https//internal.mysite.io/features/widget`. - In Fastly, create a backend called `api_host` pointing to your GrowthBook API Host. For GrowthBook Cloud customers, this will be `https://cdn.growthbook.io`. Link this backend to your Compute service. - In Fastly, create one or more backends pointing to your site origins. This includes both your main site origin URL as well as any origin URLs that you may redirect to in any URL Redirect experiments. Link these backends to your Compute service. In your request handler, you must pass these backends via the **config** parameter, as shown in the example code in the [previous doc section](#implement-the-edge-app-request-handler). - The API Host backend should be set via `config.apiHostBackend = "api_host"`. - The site origin backends are defined as an object mapping each origin URL to its corresponding backend name. They should be set via `config.backends = { "https://internal.mysite.io": "my_site" }`. ### Configure the Edge App Use a combination of environment variables and optional runtime configuration to add required fields and to customize the Edge App behavior. #### Environment variables We suggest using a [Fastly Config store ](https://docs.fastly.com/en/guides/working-with-config-stores) to set your environment variables. Create a Config store called env_vars from the Fastly dashboard and link it to your Compute service. Then, at minimum, add these required key/value pairs: ```bash PROXY_TARGET="https://internal.mysite.io" # The non-edge URL to your website GROWTHBOOK_API_HOST="https://cdn.growthbook.io" GROWTHBOOK_CLIENT_KEY="sdk-abc123" GROWTHBOOK_DECRYPTION_KEY="key_abc123" # Only include for encrypted SDK Connections ``` You may want to further customize the app. Here is a list of common customization variables: ```bash # Disable or change the rendering behavior of Visual Experiments: # ========== RUN_VISUAL_EDITOR_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "everywhere" # URL Redirect Experiments are disabled on edge by default. Because the URL does not change, some sites # may experience problems with loading assets or endpoints with relative paths: # ========== RUN_URL_REDIRECT_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "browser" RUN_CROSS_ORIGIN_URL_REDIRECT_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "browser" # Mutate browser URL via window.history.replaceState() to reflect the new URL: INJECT_REDIRECT_URL_SCRIPT="true" # default "true". # Do not inject a bootstrapped JavaScript SDK onto the page: # ========== DISABLE_INJECTIONS="true" # default "false" # Customize the edge or injected browser SDK behavior: # ========== ENABLE_STREAMING="true" # default "false". Streaming SSE updates on browser. ENABLE_STICKY_BUCKETING="true" # default "false". Use cookie-based sticky bucketing on edge and browser. ``` #### Runtime configuration You may want to provide context to your edge app at runtime rather than using environment variables. For example, if you have additional [targeting attributes](/lib/js#attributes) available, you may inject them by modifying your request handler code: ```ts /// import { ConfigStore } from "fastly:config-store"; import { gbHandleRequest, getConfigEnvFromStore } from "@growthbook/edge-fastly"; addEventListener("fetch", (event) => event.respondWith(handleRequest(event))); async function handleRequest(event) { const envVarsStore = new ConfigStore("env_vars"); const env = getConfigEnvFromStore(envVarsStore); const cookie = parse(event.request.headers.get("Cookie") || ""); const config = { // custom targeting attributes: attributes: { userType: cookie["userId"] ? "logged in" : "anonymous" }, // backends: apiHostBackend: "api_host", // Name of Fastly backend pointing to your GrowthBook API Endpoint backends: { "https://internal.mysite.io": "my_site" }, // Map of proxy origins to named Fastly backends }; return await gbHandleRequest(event.request, env, config); } ``` #### More customization options For a full list of customizations, view our vendor-agnostic [Edge Utility repository ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-utils). ### Set up a Payload Cache You can configure GrowthBook payload caching by using a [Fastly KV ](https://docs.fastly.com/en/guides/working-with-kv-stores) store. This eliminates network requests from your edge to GrowthBook which speeds up page delivery while reducing network costs. You may configure the Fastly Edge App to use either webhook-based or just-in-time payload caching (or both) depending on how you've set up your KV namespaces and SDK Webhooks. More information about setting up your payload cache can be found in the [Payload Caching with Fastly KV Store](#payload-caching-with-fastly-kv-store) doc section below. ### Tracking Experiment Views Running A/B tests requires a [tracking callback](/lib/js#experimentation-ab-testing). Our turnkey Edge App defaults to using built-in front-end tracking. The tracking call automatically integrates with Segment.io, GA4, and Google Tag Manager by using the mechanism outlined in our [HTML Script Tag](/lib/script-tag#tracking-experiment-views). In order to do this, the app keeps track of tracking calls triggered on edge and injects them into the front-end SDK to be automatically triggered on page load. You may wish to either customize front-end tracking or switch to edge tracking (or use both concurrently if running hybrid edge + front-end experiments). Why might you be interested in tracking on edge? Tracking on an edge or backend environment allows you to ensure the callback is fired before any differentiation across variations, eliminating experimental bias. While not eliminating this risk, the default injected front-end tracking introduced by our Edge App does reduce this risk relative to solely using a front-end SDK. To change the front-end tracking callback, set the `GROWTHBOOK_TRACKING_CALLBACK` to your custom tracking JS code: ```bash # todo: replace with your own tracking library GROWTHBOOK_TRACKING_CALLBACK="(experiment, results) => { console.log('browser tracking callback', {experiment, results}); }" ``` To track on edge, you must inject your own tracking callback into the edge request handler code. Any experiments that run on edge will use the edge tracking callback and not the front-end callback (hybrid edge + front-end experiments being an exception): ```ts /// import { ConfigStore } from "fastly:config-store"; import { gbHandleRequest, getConfigEnvFromStore } from "@growthbook/edge-fastly"; addEventListener("fetch", (event) => event.respondWith(handleRequest(event))); async function handleRequest(event) { const envVarsStore = new ConfigStore("env_vars"); const env = getConfigEnvFromStore(envVarsStore); const config = { edgeTrackingCallback: (experiment, results) => { // todo: replace with your tracking library console.log('edge tracking callback', {experiment, results}); }, // backends: apiHostBackend: "api_host", // Name of Fastly backend pointing to your GrowthBook API Endpoint backends: { "https://internal.mysite.io": "my_site" }, // Map of proxy origins to named Fastly backends }; return await gbHandleRequest(event.request, env, config); } ``` ### Targeting Attributes The following targeting attributes are set automatically by the Edge App. - `id` - a long-lived user identifier, stored in a `gbuuid` cookie (see [Cookie Policy and GDPR](#cookie-policy-and-gdpr) for how the cookie is written and when to use `PERSIST_UUID`) - `url` - `path` - `host` - `query` - `pageTitle` - `deviceType` - either `mobile` or `desktop` - `browser` - one of `chrome`, `edge`, `firefox`, `safari`, or `unknown` - `utmSource` - `utmMedium` - `utmCampaign` - `utmTerm` - `utmContent` You can customize both the primary identifier name (`id`) and cookie name (`gbuuid`) by setting the `UUID_KEY` and `UUID_COOKIE_NAME` environment variables respectively. As shown in the [runtime configuration](#runtime-configuration) section above, you can also pass custom attributes via runtime config. You can also skip automatic attribute generation and rely solely on custom attributes by setting the environment variable `SKIP_AUTO_ATTRIBUTES="true"`. ### Routing By default, the Edge App will process all `GET` requests (other HTTP verbs are proxied through without running through our app logic). There may be situations when you will need to provide fine-grained routing / URL targeting rules within our Edge App. You will need to include a JSON encoded string of route rules in your `ROUTES` environment variable. For instance, you may want to do a proxy pass-through (do not process) for `mysite.io/account/*` or `mysite.io/settings/*`. Your routes may look like this: ```bash ROUTES='[{ "pattern":"mysite.io/account/*", "behavior":"proxy" }, { "pattern":"mysite.io/settings/*", "behavior":"proxy" }]' ``` A route uses the following interface, with many of the properties being optional: ```ts { pattern: string; type?: "regex" | "simple"; // default: "simple" behavior?: "intercept" | "proxy" | "error"; // default: "intercept" includeFileExtensions?: boolean; // Include requests to filenames like "*.jpg". default: false (pass-through). statusCode?: number; // Alter the status code (default is 404 when using "error") body?: string; // Alter the body (for setting an error message body) } ``` When multiple routes are included in your `ROUTES` array, only the first match is used. ### Cookie Policy and GDPR The Edge App uses a random unique identifier, stored in a first-party cookie named `gbuuid`, to provide a consistent user experience to your visitors by preventing them from being re-bucketed into different A/B test variations. It follows the same mechanism as discussed in our [HTML Script Tag docs](/lib/script-tag#cookie-policy-and-gdpr). #### Persisting the `gbuuid` Cookie from the Edge By default, the Edge App generates a `gbuuid` for each visitor and exposes it as the `id` targeting attribute, but it does **not** write the cookie from the edge server itself. Instead, the cookie is persisted in the browser by the injected front-end SDK once the page loads. This works for most setups. However, the `gbuuid` cookie will **not** be written if any of the following are true: - You have disabled SDK injection (`DISABLE_INJECTIONS="true"`). - You need the `gbuuid` cookie to be available on the very first request — for example, before the browser has a chance to execute the injected SDK. In these cases, add `PERSIST_UUID="true"` to your Fastly Config store (`env_vars`). The Edge App will then include a `Set-Cookie: gbuuid=...` header on its response, ensuring the cookie is stored on the first request. If you are using `NO_AUTO_COOKIES` to wait for consent (see the [next section](#delay-storing-the-cookie-until-consent-is-granted)), no cookies — including `gbuuid` — will be written until the `growthbookpersist` event is dispatched, regardless of `PERSIST_UUID`. #### Delay Storing the Cookie Until Consent is Granted If you must delay persisting the `gbuuid` cookie until a user consents, you can set the environment variable `NO_AUTO_COOKIES="true"`. This will still generate a UUID for the user, but will not persist it. That means, if the user refreshes the page, they will have a new random UUID generated.environment You have the option to manually persist this cookie at any time, for example when a user grants consent on your cookie banner. All you need to do is fire this custom event from javascript on the rendered page: ```js document.dispatchEvent(new CustomEvent("growthbookpersist")); ``` :::note Sticky Bucketing and consent If you are using Sticky Bucketing, a persistent sticky bucket assignments cookie will automatically be generated. If you require user permission before writing cookies, you should: - Either do not enable Sticky Bucketing on edge (do not use `ENABLE_STICKY_BUCKETING`) - Or only enable Sticky Bucketing per each user via runtime configuration. (only pass `config.enableStickyBucketing: true` if user has consented — identifiable by checking for presence of the `gbuuid` cookie). ::: ### Lifecycle hooks You can perform custom logic and optionally return a response at various stages in the Edge App's lifecycle. This allows for expressiveness of custom routing, user attribute mutation, header and body (DOM) mutation, and custom feature flag and experiment implementations – while preserving the ability to automatically run Visual and URL Redirect experiments and SDK hydration. With each hook, you may mutate any of the provided attributes _or_ return an early response to halt the Edge App processing. The following hooks are available: - `onRequest` - Fired on initial user request. Can exit early based on requested URL. - `onRoute` - Fired after standard routing has been processed. Can exit early (proxy) based on manual routing logic. - `onUserAttributes` - Fired after auto-attributes have been assigned to the user. Either enhance the provided `attributes` object or exit early if desired. - `onGrowthBookInit` - Fired after the Edge App's internal GrowthBook SDK has been initialized. Call SDK functions or exit early if desired. - `onBeforeOriginFetch` - Similar hook to the above; triggers after any URL Redirect experiments have run but before any origin requests have been made. - `onOriginFetch` - Fired immediately after the origin fetch has been made, but before the full response body has been captured. Useful for exiting early based on response status or headers. - `onBodyReadyParams` - Fired once the entire response body has been parsed. In addition to early exiting, you may begin to mutate the final response body via `resHeaders` and the `setBody()` method. The text `body` as well as the optional parsed virtual DOM `root` (disabled by default, use `ALWAYS_PARSE_DOM` to enable) are exposed. NOTE: If mutating the `root` DOM, it is your responsibility to `setBody()` with the latest changes before the response is returned. - `onBeforeResponse` - The final hook fired before the response is returned to the user, triggering after both visual editor changes and client SDK hydration have been injected. While the virtual DOM is no longer available, this hook can be used to apply any final changes the body via `setBody()`. To use one or more lifecycle hooks, pass any hooks to your `gbHandleRequest` method: ```ts const hooks = { onRoute: (params) => { if (params.requestUrl === "https://mysite.io/skip") { return params.context.helpers.proxyRequest(context, params.req, params.res); } }, onBeforeResponse: (params) => { params.setBody(params.body + `console.log("custom logic")`); } }; return await gbHandleRequest(event.request, env, config, hooks); ``` ## Manual SDK Integration on Edge You may be interested in building your own edge application using the GrowthBook SDK and not using our turnkey Edge App. Or you may want to do custom feature flagging on specific routes while running our Edge App on other routes. To use the GrowthBook on edge, simply include our standard [JavaScript SDK](/lib/js) (`@growthbook/growthbook` NPM package). You will likely need to monkey-patch our SDK's built-in fetch calls in order to specify a Fastly backend. In our `@growthbook/edge-fastly` NPM package, we export a few Fastly-specific utility functions to simplify SDK payload caching (we discuss payload caching strategies in the subsequent doc section). ```ts /// import { KVStore } from "fastly:kv-store"; import { GrowthBook, setPolyfills, helpers } from "@growthbook/growthbook"; import { getPayloadFromKV, getKVLocalStoragePolyfill } from "@growthbook/edge-fastly"; addEventListener("fetch", (event) => event.respondWith(handleRequest(event))); async function handleRequest(event) { // 1. Monkey-patch the GrowthBook SDK to support Fastly backends helpers.fetchFeaturesCall = ({ host, clientKey, headers }) => fetch( `${host}/api/features/${clientKey}`, { headers, backend: config.apiHostBackend } ); // 2. Init the GrowthBook SDK and choose an optional caching strategy // A. Use the KV as a managed payload store to eliminate SDK requests to the GrowthBook API entirely. // Requires setting up an SDK Webhook. const payloadKVStore = new KVStore("gb_payload"); const payload = await getPayloadFromKV(payloadKVStore); const growthbook = new GrowthBook(gbContext); await growthbook.init({ payload: payload }); // B. Or provide a KV cache layer so that the GrowthBook SDK doesn't need to make as many requests // to the GrowthBook API. No SDK Webhook needed. const cacheKVStore = new KVStore("gb_cache"); const localStoragePolyfill = getKVLocalStoragePolyfill(cacheKVStore); setPolyfills({ localStorage: localStoragePolyfill }); await growthbook.init(); // 3. Start feature flagging if (growthbook.isOn("my-feature")) { return new Response("foo"); } return new Response("bar"); } ``` ## Payload Caching with Fastly KV Store By default, the Edge App will make a network request to the GrowthBook API on each user request in order to fetch the current feature and experiment values. This is a blocking call that delays page delivery. There is an in-memory short-lived cache layer on this call, but it won't always protect you. Convenient solutions this problem are realized through [Fastly KV ](https://docs.fastly.com/en/guides/working-with-kv-stores), an on-edge key-val store which we can leverage for persistent payload caching. There are 2 levels of KV integration available: 1. You can either completely eliminate the blocking call to the GrowthBook API by implementing a GrowthBook-to-Fastly-KV push model via **SDK Webhooks**. 2. Alternatively, you can eliminate most of these network requests by using Fastly KV as a just-in-time payload cache. You can also use either of these strategies in your own manual SDK integration via the `getPayloadFromKV` and `getKVLocalStoragePolyfill` utility functions. ### Configuring the KV store Create a Fastly KV store for your worker to interface with. Using the Fastly dashboard, create a Fastly KV store for either push-based or just-in-time payload cache (or use both if you like). By convention, we suggest naming a push-based KV store as `gb_payload` and naming a just-in-time KV store as `gb_cache`. Link your KV store(s) to your Compute service. If you are using our turnkey Edge App, you simply need to instantiate your KVStore(s) and pass them into your request handler via the **config** parameter. The Edge App will automatically use these KV stores as persistent cache if present. ```ts /// import { ConfigStore } from "fastly:config-store"; import { KVStore } from "fastly:kv-store"; import { gbHandleRequest, getConfigEnvFromStore } from "@growthbook/edge-fastly"; addEventListener("fetch", (event) => event.respondWith(handleRequest(event))); async function handleRequest(event) { const envVarsStore = new ConfigStore("env_vars"); const env = getConfigEnvFromStore(envVarsStore); const config = { apiHostBackend: "api_host", backends: { "https://internal.mysite.io": "my_site" }, gbCacheStore: new KVStore("gb_cache"), // just-in-time payload cache gbPayloadStore: new KVStore("gb_payload"), // push-based payload cache }; return await gbHandleRequest(event.request, env, config); } ``` ### Configuring a SDK Webhook For KV stored payloads (1), we eliminate network requests from edge to GrowthBook by using a GrowthBook SDK Webhook to push the SDK payload to the KV store on change. 1. Create an [SDK Webhook](/app/webhooks/sdk-webhooks) on the same SDK Connection that you are using for edge integration. You do not need to worry about the receiving end of the webhook (verifying GrowthBook signatures, etc). 2. Select **Fastly KV** as the Webhook Type and fill out the following fields: - Store ID - Key - API Token Now whenever feature and experiment values change, your Fastly worker will have immediate access to the latest values. You can also test the webhook by using the "Test Webhook" button on the SDK Connection page. Under the hood, the webhook is being configured with the following properties. If you need to change any of these settings for any reason, you can always edit the webhook. - **Endpoint URL** is being set to ``` https://api.fastly.com/resources/stores/kv/{store_id}/keys/{key} ``` - **Method** is being set to `PUT` - A **Fastly-Key Header** is being added with your API Token - The **Payload format** is being set to `SDK Payload only` ## Supported Features --- ## Lambda@Edge Edge App & SDK import ExternalLink from '@site/src/components/ExternalLink'; import Pill from '@site/src/components/Pill'; import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Lambda@Edge Edge App & SDK beta :::caution Beta Our Lambda@Edge implementation is in **beta**. If you experience any issues, let us know either on [Slack](https://slack.growthbook.io/) or [create an issue](https://github.com/growthbook/growthbook-proxy/issues). ::: ## Overview GrowthBook currently supports two levels of integration with most edge workers, including Lambda@Edge: 1. Our turnkey Edge App - Automatically run server-side or hybrid [Visual Experiments](/app/visual) without redraw flicker. - Automatically run server-side or hybrid [URL Redirect Experiments](/app/url-redirects) without flicker or delay. - Perform custom feature flagging and experimentation logic. - Optionally inject the JavaScript SDK with hydrated payload, allowing the front-end to pick up where the edge left off without any extra network requests. We use an enhanced version of our [HTML Script Tag](/lib/script-tag) for this purpose. 2. Support for edge apps using our JavaScript SDK ## References - Our Lambda@Edge SDK repository, which supports the above use cases, is [here ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-lambda) - You may find it useful to review our [JavaScript SDK](/lib/js). Many of the concepts which apply to both on-edge and injected frontend SDKs are based on our JS SDK. ## Worker Configuration :::tip Get up to speed with Lambda@Edge This tutorial assumes some familiarity with building and deploying AWS Lambda@Edge applications. You can get up to speed by following the AWS [Tutorial: Create a basic Lambda@Edge function ](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-edge-how-it-works-tutorial.html) guide. Note that our Edge App responds directly to a `viewer-request` without forwarding to an origin; interaction with CloudFront is minimal (Step 2 in the AWS tutorial). ::: You may either use our turnkey Edge App for Lambda@Edge or build your own app from scratch using our JavaScript SDK. ## Turnkey Edge App Our Edge App runs as a smart proxy layer between your application and your end users. In absence of Visual or URL Redirect experiments, the Edge App will simply proxy the user request to your site and return the response, optionally injecting a fully-bootstrapped JavaScript SDK onto the rendered HTML page. If the request URL matches an Visual or URL Redirect experiment and the targeting conditions are satisfied, the Edge App may also perform one or more URL redirects behind the scenes (the public-facing URL does not change) and/or mutate the DOM for Visual Experiments. Additionally, by using [lifecycle hooks](#lifecycle-hooks) you can perform custom logic such as feature flagging as well as proxying and early returns. :::caution Body size constraints `viewer-request` responses are currently limited to **40 KB**! Please ensure the DOM returned by both your origin server and by your Lambda function are optimized. You also may need to skip SDK injection when using Lambda@Edge. ::: :::note URL Redirects on edge The Edge App defaults to running URL Redirect Experiments in the browser only. This is because edge redirects load a separate page's content without altering the URL. After the redirect, some sites may experience problems with loading assets or endpoints with relative paths. You can enable URL Redirects on edge by setting environment variable `RUN_URL_REDIRECT_EXPERIMENTS` to "edge" or "everywhere". Additionally if your redirect is cross-domain (e.g. redirection from "public.mysite.io" to "newsite.io"), you must also set `RUN_CROSS_ORIGIN_URL_REDIRECT_EXPERIMENTS`. See [environment variables](#environment-variables) for more information. ::: We will assume that you have a basic Lambda@Edge application set up, and that it is configured to respond to viewer requests. Note that our Edge App will not attempt to reach your CF cache nor origin and instead will fetch/proxy to your web server during the _viewer request / response_ lifecycle. Therefore a minimal CloudFront setup is advised. Once your application is set up, simply install the SDK and implement our custom request handler. ### Install the SDK ```bash npm2yarn npm install --save @growthbook/edge-lambda ``` ### Implement the Edge App request handler A basic implementation of our Edge App only requires a few lines of code: ```ts import { handleRequest } from "@growthbook/edge-lambda"; export async function handler(event, ctx, callback) { // Manually build your environment: const env = buildEnv(); // Specify additional edge endpoint information: env.host = "www.mysite.io"; // Uncomment for ease of testing locally - returns response instead of using callback(): // env.returnResponse = true; handleRequest(event, callback, env); } function buildEnv() { // todo: define environment variables here } ``` ### Configure the Edge App Use a combination of environment variables and optional runtime configuration to add required fields and to customize the Edge App behavior. #### Environment variables Unfortunately Lambda@Edge does not have native support for environment variables. You will need to implement your own mechanism to build an `Env` environment (a TypeScript interface is exported in `"@growthbook/edge-lambda"`). You will need to inject your environment variables into the handler either directly into your codebase or at compile time. Our `buildEnv()` method is a placeholder for your preferred mechanism. :::tip Tip (for a subset of use cases) We do offer a utility function called `mapHeadersToConfigEnv(req, originType="custom", prefix="x-env-")` (exported from `"@growthbook/edge-lambda"`) that can build a valid `Env` environment from custom CF origin headers. However this is only useful if your app is set up to reach your origin server through CF (this is not the default behavior of our Edge App). ::: Add these required fields, at minimum, to your environment variables: ```typescript function buildEnv() { return { // required fields: PROXY_TARGET: "https://internal.mysite.io", // The non-edge URL to your website GROWTHBOOK_API_HOST: "https://cdn.growthbook.io", GROWTHBOOK_CLIENT_KEY: "sdk-abc123", GROWTHBOOK_DECRYPTION_KEY: "key_abc123", // Only include for encrypted SDK Connections }; } ``` You may want to further customize the app. Here is a list of common customization variables: ```bash # Disable or change the rendering behavior of Visual Experiments: # ========== RUN_VISUAL_EDITOR_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "everywhere" # URL Redirect Experiments are disabled on edge by default. Because the URL does not change, some sites # may experience problems with loading assets or endpoints with relative paths: # ========== RUN_URL_REDIRECT_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "browser" RUN_CROSS_ORIGIN_URL_REDIRECT_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "browser" # Mutate browser URL via window.history.replaceState() to reflect the new URL: INJECT_REDIRECT_URL_SCRIPT="true" # default "true". # Do not inject a bootstrapped JavaScript SDK onto the page: # ========== DISABLE_INJECTIONS="true" # default "false" # Customize the edge or injected browser SDK behavior: # ========== ENABLE_STREAMING="true" # default "false". Streaming SSE updates on browser. ENABLE_STICKY_BUCKETING="true" # default "false". Use cookie-based sticky bucketing on edge and browser. ``` #### Runtime configuration You may want to provide context to your edge app at runtime rather than using environment variables. For example, if you have additional [targeting attributes](/lib/js#attributes) available, you may inject them by modifying your request handler code: ```ts import { handleRequest } from "@growthbook/edge-lambda"; import { getCookies } from "./helpers"; export async function handler(event, ctx, callback) { const env = buildEnv(); env.host = "www.mysite.io"; // example getCookies method const cookie = getCookies(event); const config = { attributes: { userType: cookie["userId"] ? "logged in" : "anonymous" } }; handleRequest(event, callback, env, config); } ``` #### More customization options For a full list of customizations, view our vendor-agnostic [Edge Utility repository ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-utils). ### Tracking Experiment Views Running A/B tests requires a [tracking callback](/lib/js#experimentation-ab-testing). Our turnkey Edge App defaults to using built-in front-end tracking. The tracking call automatically integrates with Segment.io, GA4, and Google Tag Manager by using the mechanism outlined in our [HTML Script Tag](/lib/script-tag#tracking-experiment-views). In order to do this, the app keeps track of tracking calls triggered on edge and injects them into the front-end SDK to be automatically triggered on page load. You may wish to either customize front-end tracking or switch to edge tracking (or use both concurrently if running hybrid edge + front-end experiments). Why might you be interested in tracking on edge? Tracking on an edge or backend environment allows you to ensure the callback is fired before any differentiation across variations, eliminating experimental bias. While not eliminating this risk, the default injected front-end tracking introduced by our Edge App does reduce this risk relative to solely using a front-end SDK. To change the front-end tracking callback, set the `GROWTHBOOK_TRACKING_CALLBACK` to your custom tracking JS code: ```bash # todo: replace with your own tracking library GROWTHBOOK_TRACKING_CALLBACK="(experiment, results) => { console.log('browser tracking callback', {experiment, results}); }" ``` To track on edge, you must inject your own tracking callback into the edge request handler code. Any experiments that run on edge will use the edge tracking callback and not the front-end callback (hybrid edge + front-end experiments being an exception): ```ts import { handleRequest } from "@growthbook/edge-lambda"; export async function handler(event, ctx, callback) { const env = buildEnv(); env.host = "www.mysite.io"; const config = { edgeTrackingCallback: (experiment, results) => { // todo: replace with your tracking library console.log('edge tracking callback', {experiment, results}); } }; handleRequest(event, callback, env, config); } ``` ### Targeting Attributes The following targeting attributes are set automatically by the Edge App. - `id` - a long-lived user identifier, stored in a `gbuuid` cookie (see [Cookie Policy and GDPR](#cookie-policy-and-gdpr) for how the cookie is written and when to use `PERSIST_UUID`) - `url` - `path` - `host` - `query` - `pageTitle` - `deviceType` - either `mobile` or `desktop` - `browser` - one of `chrome`, `edge`, `firefox`, `safari`, or `unknown` - `utmSource` - `utmMedium` - `utmCampaign` - `utmTerm` - `utmContent` You can customize both the primary identifier name (`id`) and cookie name (`gbuuid`) by setting the `UUID_KEY` and `UUID_COOKIE_NAME` environment variables respectively. As shown in the [runtime configuration](#runtime-configuration) section above, you can also pass custom attributes via runtime config. You can also skip automatic attribute generation and rely solely on custom attributes by setting the environment variable `SKIP_AUTO_ATTRIBUTES="true"`. ### Routing By default, the Edge App will process all `GET` requests (other HTTP verbs are proxied through without running through our app logic). It is generally preferable to configure your routing rules outside of our Edge App when possible. For instance, you may only want to invoke the Edge App at `https://yourdomain.io/landing-page`. There may be situations when you will need to provide finer-grained routing / URL targeting rules within our Edge App. You will need to include a JSON encoded string of route rules in your `ROUTES` environment variable. For instance, you may want to do a proxy pass-through (do not process) for `mysite.io/account/*` or `mysite.io/settings/*`. Your routes may look like this: ```bash ROUTES='[{ "pattern":"mysite.io/account/*", "behavior":"proxy" }, { "pattern":"mysite.io/settings/*", "behavior":"proxy" }]' ``` A route uses the following interface, with many of the properties being optional: ```ts { pattern: string; type?: "regex" | "simple"; // default: "simple" behavior?: "intercept" | "proxy" | "error"; // default: "intercept" includeFileExtensions?: boolean; // Include requests to filenames like "*.jpg". default: false (pass-through). statusCode?: number; // Alter the status code (default is 404 when using "error") body?: string; // Alter the body (for setting an error message body) } ``` When multiple routes are included in your `ROUTES` array, only the first match is used. ### Cookie Policy and GDPR The Edge App uses a random unique identifier, stored in a first-party cookie named `gbuuid`, to provide a consistent user experience to your visitors by preventing them from being re-bucketed into different A/B test variations. It follows the same mechanism as discussed in our [HTML Script Tag docs](/lib/script-tag#cookie-policy-and-gdpr). #### Persisting the `gbuuid` Cookie from the Edge By default, the Edge App generates a `gbuuid` for each visitor and exposes it as the `id` targeting attribute, but it does **not** write the cookie from the edge server itself. Instead, the cookie is persisted in the browser by the injected front-end SDK once the page loads. This works for most setups. However, the `gbuuid` cookie will **not** be written if any of the following are true: - You have disabled SDK injection (`DISABLE_INJECTIONS="true"`), which is common with Lambda@Edge due to the 40 KB `viewer-request` body limit. - You need the `gbuuid` cookie to be available on the very first request — for example, before the browser has a chance to execute the injected SDK. In these cases, add `PERSIST_UUID: "true"` to the environment returned by your `buildEnv()` function. The Edge App will then include a `Set-Cookie: gbuuid=...` header on its response, ensuring the cookie is stored on the first request. ```typescript function buildEnv() { return { PROXY_TARGET: "https://internal.mysite.io", GROWTHBOOK_API_HOST: "https://cdn.growthbook.io", GROWTHBOOK_CLIENT_KEY: "sdk-abc123", PERSIST_UUID: "true", }; } ``` If you are using `NO_AUTO_COOKIES` to wait for consent (see the [next section](#delay-storing-the-cookie-until-consent-is-granted)), no cookies — including `gbuuid` — will be written until the `growthbookpersist` event is dispatched, regardless of `PERSIST_UUID`. #### Delay Storing the Cookie Until Consent is Granted If you must delay persisting the `gbuuid` cookie until a user consents, you can set the environment variable `NO_AUTO_COOKIES="true"`. This will still generate a UUID for the user, but will not persist it. That means, if the user refreshes the page, they will have a new random UUID generated.environment You have the option to manually persist this cookie at any time, for example when a user grants consent on your cookie banner. All you need to do is fire this custom event from javascript on the rendered page: ```js document.dispatchEvent(new CustomEvent("growthbookpersist")); ``` :::note Sticky Bucketing and consent If you are using Sticky Bucketing, a persistent sticky bucket assignments cookie will automatically be generated. If you require user permission before writing cookies, you should: - Either do not enable Sticky Bucketing on edge (do not use `ENABLE_STICKY_BUCKETING`) - Or only enable Sticky Bucketing per each user via runtime configuration. (only pass `config.enableStickyBucketing: true` if user has consented — identifiable by checking for presence of the `gbuuid` cookie). ::: ### Lifecycle hooks You can perform custom logic and optionally return a response at various stages in the Edge App's lifecycle. This allows for expressiveness of custom routing, user attribute mutation, header and body (DOM) mutation, and custom feature flag and experiment implementations – while preserving the ability to automatically run Visual and URL Redirect experiments and SDK hydration. With each hook, you may mutate any of the provided attributes _or_ return an early response to halt the Edge App processing. The following hooks are available: - `onRequest` - Fired on initial user request. Can exit early based on requested URL. - `onRoute` - Fired after standard routing has been processed. Can exit early (proxy) based on manual routing logic. - `onUserAttributes` - Fired after auto-attributes have been assigned to the user. Either enhance the provided `attributes` object or exit early if desired. - `onGrowthBookInit` - Fired after the Edge App's internal GrowthBook SDK has been initialized. Call SDK functions or exit early if desired. - `onBeforeOriginFetch` - Similar hook to the above; triggers after any URL Redirect experiments have run but before any origin requests have been made. - `onOriginFetch` - Fired immediately after the origin fetch has been made, but before the full response body has been captured. Useful for exiting early based on response status or headers. - `onBodyReadyParams` - Fired once the entire response body has been parsed. In addition to early exiting, you may begin to mutate the final response body via `resHeaders` and the `setBody()` method. The text `body` as well as the optional parsed virtual DOM `root` (disabled by default, use `ALWAYS_PARSE_DOM` to enable) are exposed. NOTE: If mutating the `root` DOM, it is your responsibility to `setBody()` with the latest changes before the response is returned. - `onBeforeResponse` - The final hook fired before the response is returned to the user, triggering after both visual editor changes and client SDK hydration have been injected. While the virtual DOM is no longer available, this hook can be used to apply any final changes the body via `setBody()`. To use one or more lifecycle hooks, pass any hooks to your `handleRequest` method: ```ts const hooks = { onRoute: (params) => { if (params.requestUrl === "https://mysite.io/skip") { return params.context.helpers.proxyRequest(context, params.req, params.res); } }, onBeforeResponse: (params) => { params.setBody(params.body + `console.log("custom logic")`); } }; handleRequest(event, callback, env, undefined, hooks); ``` ## Manual SDK Integration on Edge You may be interested in building your own edge application using the GrowthBook SDK and not using our turnkey Edge App. Or you may want to do custom feature flagging on specific routes while running our Edge App on other routes. To use the GrowthBook on edge, simply include our standard [JavaScript SDK](/lib/js) (`@growthbook/growthbook` NPM package). ```ts import { GrowthBook, setPolyfills } from "@growthbook/growthbook"; export async function handler(event, ctx, callback) { // 1. Init the GrowthBook SDK and choose an optional caching strategy // A. Use the KV as a managed payload store to eliminate SDK requests to the GrowthBook API entirely. // Requires setting up an SDK Webhook. const payload = await getPayloadFromProvider(); // not implemented, build your own const growthbook = new GrowthBook(gbContext); await growthbook.init({ payload: payload }); // B. Or provide a KV cache layer so that the GrowthBook SDK doesn't need to make as many requests // to the GrowthBook API. No SDK Webhook needed. const localStoragePolyfill = getLocalStoragePolyfill(env); // not implemented, build your own setPolyfills({ localStorage: localStoragePolyfill }); await growthbook.init(); // 2. Start feature flagging if (growthbook.isOn("my-feature")) { const resp = { status: "200", body: "foo" }; callback(null, resp); } else { const resp = { status: "200", body: "bar" }; callback(null, resp); } } ``` ## Payload Caching via edge datastore By default, the Edge App will make a network request to the GrowthBook API on each user request in order to fetch the current feature and experiment values. This is a blocking call that delays page delivery. There is an in-memory short-lived cache layer on this call, but it won't always protect you. If you have access to a distributed key-value store such as DynamoDB, you can likely overcome this problem. There are 2 levels of key-value integration available: 1. You can either completely eliminate the blocking call to the GrowthBook API by implementing a GrowthBook-to-edge-keyval push model via **SDK Webhooks**. 2. Alternatively, you can eliminate most of these network requests by using an edge key-val store as a just-in-time payload cache. You can also use these strategies in your own manual SDK integration. We are unable to offer specific guidance about how to configure or connect to your key-val store because there are many possible network configurations and data stores within an AWS edge application. ### Configuring a SDK Webhook For key-val stored payloads (1), we eliminate network requests from edge to GrowthBook by using a GrowthBook SDK Webhook to push the SDK payload to the key-val store on change. 1. Create an [SDK Webhook](/app/webhooks/sdk-webhooks) on the same SDK Connection that you are using for edge integration. 2. Select **HTTP Endpoint** as the Webhook Type. 3. Set the **Endpoint URL** to your key-val store's REST API endpoint, if available. You may need to build your own private Lambda endpoint to handle the webhook, in which case webhook verification may be important. 4. Change the **Method** to `PUT` (or whichever verb is required by your endpoint). 5. Set the **Payload format** to "SDK Payload only". Now whenever feature and experiment values change, your edge worker will have immediate access to the latest values. You can also test the webhook by using the "Test Webhook" button on the SDK Connection page. ## Supported Features --- ## GrowthBook Edge App for other edge providers import ExternalLink from '@site/src/components/ExternalLink'; import Pill from '@site/src/components/Pill'; import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # GrowthBook Edge App for other providers ## Overview GrowthBook currently supports two levels of integration with most edge workers: 1. Our turnkey Edge App - Automatically run server-side or hybrid [Visual Experiments](/app/visual) without redraw flicker. - Automatically run server-side or hybrid [URL Redirect Experiments](/app/url-redirects) without flicker or delay. - Perform custom feature flagging and experimentation logic. - Optionally inject the JavaScript SDK with hydrated payload, allowing the front-end to pick up where the edge left off without any extra network requests. We use an enhanced version of our [HTML Script Tag](/lib/script-tag) for this purpose. 2. Support for edge apps using our JavaScript SDK ## References - Our base Edge App repository, which supports the above use cases, is [here ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-utils) - You may find it useful to review our [JavaScript SDK](/lib/js). Many of the concepts which apply to both on-edge and injected frontend SDKs are based on our JS SDK. ## Turnkey Edge App Our Edge App runs as a smart proxy layer between your application and your end users. In absence of Visual or URL Redirect experiments, the Edge App will simply proxy the user request to your site and return the response, optionally injecting a fully-bootstrapped JavaScript SDK onto the rendered HTML page. If the request URL matches an Visual or URL Redirect experiment and the targeting conditions are satisfied, the Edge App may also perform one or more URL redirects behind the scenes (the public-facing URL does not change) and/or mutate the DOM for Visual Experiments. Additionally, by using lifecycle hooks you can perform custom logic such as feature flagging as well as proxying and early returns. :::note URL Redirects on edge The Edge App defaults to running URL Redirect Experiments in the browser only. This is because edge redirects load a separate page's content without altering the URL. After the redirect, some sites may experience problems with loading assets or endpoints with relative paths. You can enable URL Redirects on edge by setting environment variable `RUN_URL_REDIRECT_EXPERIMENTS` to "edge" or "everywhere". Additionally if your redirect is cross-domain (e.g. redirection from "public.mysite.io" to "newsite.io"), you must also set `RUN_CROSS_ORIGIN_URL_REDIRECT_EXPERIMENTS`. See [environment variables](#environment-variables) for more information. ::: Setting up our turnkey Edge App is usually simple for most JavaScript-based edge workers. In this guide we do not discuss bootstrapping your project, but rather how to implement our Edge App once you have a basic edge worker environment set up. ### Install the SDK ```bash npm2yarn npm install --save @growthbook/edge-utils ``` ### Implement the Edge App request handler A basic implementation of our Edge App only requires a few lines of code. Your implementation however will be vendor specific: ```ts import { edgeApp, getConfig, defaultContext } from "@growthbook/edge-utils"; // Build a request handler // note: we assume availability of `env` (environment variables) export async function handler(request, env) { const context = await init(env); return edgeApp(context, request); } // Build context for the app function init(env) { // Automatically parse environment variables into app context const context = defaultContext; context.config = getConfig(env); context.helpers = { // define utility functions for request/response manipulation }; return context; } ``` You will need to write our own set of helper methods (`context.helpers`) that correspond to how your edge provider handles various request and response utilities. ### Define helper methods You will need to build helper methods based on the following interfaces: ```typescript { // Get the full URL from the user request getRequestURL?: (req: Req) => string; // Get the request method (GET, POST, etc) getRequestMethod?: (req: Req) => string; // Get a specific header from the request getRequestHeader?: (req: Req, key: string) => string | undefined; // Construct or prepare a response for sending to the client // note: For many vendors, this function should build a new response object sendResponse?: ( ctx: Context, res?: Res, // Only needed for engines where res already exists headers?: Record, body?: string, cookies?: Record, // Optionally use ctx.helpers.setCookie(resp, key, cookies[key]) to assign. status?: number, ) => unknown; // Fetch an external resource from the edge app // note: For many vendors, you can ignore ctx fetch?: (ctx: Context, url: string) => Promise; // For sending a pass-through response without Edge App interception // note: For many vendors, this will map directly to helpers.fetch proxyRequest?: ( ctx: Context, req: Req, res?: Res, next?: any, ) => Promise; // Get cookie from request (for UUID and StickyBucketing persistence) getCookie?: (req: Req, key: string) => string; // Set a cookie on response (unused unless you choose to persist on edge) setCookie?: (res: Res, key: string, value: string) => void; } ``` You may reference example implementations of these methods in our [Cloudflare repository ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-cloudflare/src/helpers.ts). ### Configure the Edge App Use a combination of environment variables and optional runtime configuration to add required fields and to customize the Edge App behavior. #### Environment variables Add these required fields, at minimum, to your environment variables: ```bash PROXY_TARGET="https://internal.mysite.io" # The non-edge URL to your website GROWTHBOOK_API_HOST="https://cdn.growthbook.io" GROWTHBOOK_CLIENT_KEY="sdk-abc123" GROWTHBOOK_DECRYPTION_KEY="key_abc123" # Only include for encrypted SDK Connections ``` You may want to further customize the app. Here is a list of common customization variables: ```bash # Disable or change the rendering behavior of Visual Experiments: # ========== RUN_VISUAL_EDITOR_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "everywhere" # URL Redirect Experiments are disabled on edge by default. Because the URL does not change, some sites # may experience problems with loading assets or endpoints with relative paths: # ========== RUN_URL_REDIRECT_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "browser" RUN_CROSS_ORIGIN_URL_REDIRECT_EXPERIMENTS="everywhere"|"edge"|"browser"|"skip" # default: "browser" # Mutate browser URL via window.history.replaceState() to reflect the new URL: INJECT_REDIRECT_URL_SCRIPT="true" # default "true". # Do not inject a bootstrapped JavaScript SDK onto the page: # ========== DISABLE_INJECTIONS="true" # default "false" # Customize the edge or injected browser SDK behavior: # ========== ENABLE_STREAMING="true" # default "false". Streaming SSE updates on browser. ENABLE_STICKY_BUCKETING="true" # default "false". Use cookie-based sticky bucketing on edge and browser. ``` #### Runtime configuration You may want to provide context to your edge app at runtime rather than using environment variables. For example, if you have additional [targeting attributes](/lib/js#attributes) available, you may inject them by modifying your request handler code: ```ts import { edgeApp, getConfig } from "@growthbook/edge-utils"; export async function handler(request, env) { const context = await init(env); const userId = context.helpers.getCookie(request, "userId"); context.config.attributes.userType = userId ? "logged in" : "anonymous"; return edgeApp(context, request); } ``` #### More customization options For a full list of customizations, view our [Edge Utility repository ](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/edge-utils). ### Tracking Experiment Views Running A/B tests requires a [tracking callback](/lib/js#experimentation-ab-testing). Our turnkey Edge App defaults to using built-in front-end tracking. The tracking call automatically integrates with Segment.io, GA4, and Google Tag Manager by using the mechanism outlined in our [HTML Script Tag](/lib/script-tag#tracking-experiment-views). In order to do this, the app keeps track of tracking calls triggered on edge and injects them into the front-end SDK to be automatically triggered on page load. You may wish to either customize front-end tracking or switch to edge tracking (or use both concurrently if running hybrid edge + front-end experiments). Why might you be interested in tracking on edge? Tracking on an edge or backend environment allows you to ensure the callback is fired before any differentiation across variations, eliminating experimental bias. While not eliminating this risk, the default injected front-end tracking introduced by our Edge App does reduce this risk relative to solely using a front-end SDK. To change the front-end tracking callback, set the `GROWTHBOOK_TRACKING_CALLBACK` to your custom tracking JS code: ```bash # todo: replace with your own tracking library GROWTHBOOK_TRACKING_CALLBACK="(experiment, results) => { console.log('browser tracking callback', {experiment, results}); }" ``` To track on edge, you must inject your own tracking callback into the edge request handler code. Any experiments that run on edge will use the edge tracking callback and not the front-end callback (hybrid edge + front-end experiments being an exception): ```ts import { edgeApp, getConfig } from "@growthbook/edge-utils"; export async function handler(request, env) { const context = await init(env); context.config.edgeTrackingCallback = (experiment, results) => { // todo: replace with your tracking library console.log('edge tracking callback', {experiment, results}); } return edgeApp(context, request); } ``` ### Targeting Attributes The following targeting attributes are set automatically by the Edge App. - `id` - a long-lived user identifier, stored in a `gbuuid` cookie (see [Cookie Policy and GDPR](#cookie-policy-and-gdpr) for how the cookie is written and when to use `PERSIST_UUID`) - `url` - `path` - `host` - `query` - `pageTitle` - `deviceType` - either `mobile` or `desktop` - `browser` - one of `chrome`, `edge`, `firefox`, `safari`, or `unknown` - `utmSource` - `utmMedium` - `utmCampaign` - `utmTerm` - `utmContent` You can customize both the primary identifier name (`id`) and cookie name (`gbuuid`) by setting the `UUID_KEY` and `UUID_COOKIE_NAME` environment variables respectively. As shown in the [runtime configuration](#runtime-configuration) section above, you can also pass custom attributes via runtime config. You can also skip automatic attribute generation and rely solely on custom attributes by setting the environment variable `SKIP_AUTO_ATTRIBUTES="true"`. ### Routing By default, the Edge App will process all `GET` requests with `Content-Type: text/html` encoding (other HTTP verbs and content types are proxied through without running through our app logic). There may be situations when you will need to provide finer-grained routing / URL targeting rules within our Edge App. You will need to include a JSON encoded string of route rules in your `ROUTES` environment variable. For instance, you may want to do a proxy pass-through (do not process) for `mysite.io/account/*` or `mysite.io/settings/*`. Your routes may look like this: ```bash ROUTES='[{ "pattern":"mysite.io/account/*", "behavior":"proxy" }, { "pattern":"mysite.io/settings/*", "behavior":"proxy" }]' ``` A route uses the following interface, with many of the properties being optional: ```ts { pattern: string; type?: "regex" | "simple"; // default: "simple" behavior?: "intercept" | "proxy" | "error"; // default: "intercept" includeFileExtensions?: boolean; // Include requests to filenames like "*.jpg". default: false (pass-through). statusCode?: number; // Alter the status code (default is 404 when using "error") body?: string; // Alter the body (for setting an error message body) } ``` When multiple routes are included in your `ROUTES` array, only the first match is used. ### Cookie Policy and GDPR The Edge App uses a random unique identifier, stored in a first-party cookie named `gbuuid`, to provide a consistent user experience to your visitors by preventing them from being re-bucketed into different A/B test variations. It follows the same mechanism as discussed in our [HTML Script Tag docs](/lib/script-tag#cookie-policy-and-gdpr). #### Persisting the `gbuuid` Cookie from the Edge By default, the Edge App generates a `gbuuid` for each visitor and exposes it as the `id` targeting attribute, but it does **not** write the cookie from the edge server itself. Instead, the cookie is persisted in the browser by the injected front-end SDK once the page loads. This works for most setups. However, the `gbuuid` cookie will **not** be written if any of the following are true: - You have disabled SDK injection (`DISABLE_INJECTIONS="true"`). - You need the `gbuuid` cookie to be available on the very first request — for example, before the browser has a chance to execute the injected SDK. In these cases, set the environment variable `PERSIST_UUID="true"`. The Edge App will then call your `helpers.setCookie` implementation to write a `gbuuid` cookie on the response, ensuring the cookie is stored on the first request. Make sure your `setCookie` helper is implemented for your edge provider (see [Define helper methods](#define-helper-methods)). If you are using `NO_AUTO_COOKIES` to wait for consent (see the [next section](#delay-storing-the-cookie-until-consent-is-granted)), no cookies — including `gbuuid` — will be written until the `growthbookpersist` event is dispatched, regardless of `PERSIST_UUID`. #### Delay Storing the Cookie Until Consent is Granted If you must delay persisting the `gbuuid` cookie until a user consents, you can set the environment variable `NO_AUTO_COOKIES="true"`. This will still generate a UUID for the user, but will not persist it. That means, if the user refreshes the page, they will have a new random UUID generated.environment You have the option to manually persist this cookie at any time, for example when a user grants consent on your cookie banner. All you need to do is fire this custom event from javascript on the rendered page: ```js document.dispatchEvent(new CustomEvent("growthbookpersist")); ``` :::note Sticky Bucketing and consent If you are using Sticky Bucketing, a persistent sticky bucket assignments cookie will automatically be generated. If you require user permission before writing cookies, you should: - Either do not enable Sticky Bucketing on edge (do not use `ENABLE_STICKY_BUCKETING`) - Or only enable Sticky Bucketing per each user via runtime configuration. (only pass `config.enableStickyBucketing: true` if user has consented — identifiable by checking for presence of the `gbuuid` cookie). ::: ## Manual SDK Integration on Edge You may be interested in building your own edge application using the GrowthBook SDK and not using our turnkey Edge App. Or you may want to do custom feature flagging on specific routes while running our Edge App on other routes. To use the GrowthBook on edge, simply include our standard [JavaScript SDK](/lib/js) (`@growthbook/growthbook` NPM package). ```ts import { GrowthBook, setPolyfills } from "@growthbook/growthbook"; export default { async fetch(request) { // 1. Init the GrowthBook SDK and choose an optional caching strategy // A. Use the KV as a managed payload store to eliminate SDK requests to the GrowthBook API entirely. // Requires setting up an SDK Webhook. const payload = await getPayloadFromProvider(env); // not implemented, build your own const growthbook = new GrowthBook(gbContext); await growthbook.init({ payload: payload }); // B. Or provide a KV cache layer so that the GrowthBook SDK doesn't need to make as many requests // to the GrowthBook API. No SDK Webhook needed. const localStoragePolyfill = getLocalStoragePolyfill(env); // not implemented, build your own setPolyfills({ localStorage: localStoragePolyfill }); await growthbook.init(); // 2. Start feature flagging if (growthbook.isOn("my-feature")) { return new Response("foo"); } return new Response("bar"); } } ``` ## Payload Caching via edge datastore By default, the Edge App will make a network request to the GrowthBook API on each user request in order to fetch the current feature and experiment values. This is a blocking call that delays page delivery. There is an in-memory short-lived cache layer on this call, but it won't always protect you. If you have access to a distributed key-val store on your edge, you can likely overcome this problem. There are 2 levels of key-val integration available: 1. You can either completely eliminate the blocking call to the GrowthBook API by implementing a GrowthBook-to-edge-keyval push model via **SDK Webhooks**. 2. Alternatively, you can eliminate most of these network requests by using an edge key-val store as a just-in-time payload cache. You can also use these strategies in your own manual SDK integration. We are unable to offer specific guidance about how to configure or connect to your key-val store because the details vary heavily by vendor. For inspiration, you can see how we've addressed key-val implementation with Cloudflare Workers in our [Cloudflare docs](/lib/edge/cloudflare#payload-caching-with-cloudflare-kv-store). ### Configuring a SDK Webhook For key-val stored payloads (1), we eliminate network requests from edge to GrowthBook by using a GrowthBook SDK Webhook to push the SDK payload to the key-val store on change. 1. Create an [SDK Webhook](/app/webhooks/sdk-webhooks) on the same SDK Connection that you are using for edge integration. 2. Select **HTTP Endpoint** as the Webhook Type. 3. Set the **Endpoint URL** to your key-val provider's REST API endpoint. However, not all edge vendors will have a public REST endpoint for setting the key-val cache. You may need to build your own endpoint to handle the webhook, in which case webhook verification may be important. 4. Change the **Method** to `PUT` (or whichever verb is required by your vendor or endpoint). 5. Add any vendor-specific authorization headers. This may not be required depending on your specific vendor: ```json { "Authorization": "Bearer YOUR_REST_API_TOKEN" } ``` 6. Set the **Payload format** to "SDK Payload only". Now whenever feature and experiment values change, your edge worker will have immediate access to the latest values. You can also test the webhook by using the "Test Webhook" button on the SDK Connection page. ## Supported Features --- ## Elixir SDK import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Elixir This SDK follows the guidelines set out in [GrowthBook's documentation](/lib/build-your-own), and the API is tested on conformance with the test cases from the JS SDK. To ensure an Elixir-friendly API, the implementation deviates from the official SDK in the following ways: - Instead of tuple-lists, this library uses actual tuples - Comparisons with `undefined` are implemented by using `:undefined` - Function names are converted to `snake_case`, and `is_` prefix is replaced with a `?` suffix - Instead of classes, a Context struct is used (similar to `%Plug.Conn{}` in `plug`) ## Installation Add `growthbook` to your list of dependencies in `mix.exs`: ```elixir def deps do [ {:growthbook, "~> 0.3"} ] end ``` ## Quick Start Get started with GrowthBook in just a few steps: ```elixir # 1. Initialize GrowthBook with your API key GrowthBook.init( client_key: "sdk-abc123", api_host: "https://cdn.growthbook.io" ) # 2. Create a context with user attributes context = GrowthBook.build_context(%{ "id" => "user-123", "country" => "US" }) # 3. Evaluate features if GrowthBook.feature(context, "new-dashboard").on? do render_new_dashboard() else render_old_dashboard() end # Get feature values button_color = GrowthBook.feature(context, "button-color").value max_retries = GrowthBook.feature(context, "max-retries").value || 3 ``` That's it! GrowthBook will automatically fetch and refresh your features in the background. ## Automatic Features Refresh The Elixir SDK provides a GenServer-based feature repository that automatically fetches and caches features from the GrowthBook API. ### Initialization Options ```elixir GrowthBook.init( # Required client_key: "sdk-abc123", # Optional configuration api_host: "https://cdn.growthbook.io", # Default: "https://cdn.growthbook.io" decryption_key: "key_abc123", # For encrypted features swr_ttl_seconds: 60, # Cache TTL in seconds (default: 60) refresh_strategy: :periodic, # :periodic (default) or :manual # Optional callback on_refresh: fn features -> # Called whenever features are successfully refreshed Logger.info("Features updated: #{map_size(features)}") end ) ``` ### Refresh Strategies The SDK supports two refresh strategies: #### Periodic Refresh (Default) Features are automatically refreshed in the background based on the TTL: ```elixir GrowthBook.init( client_key: "sdk-abc123", refresh_strategy: :periodic, # Auto-refresh enabled swr_ttl_seconds: 60 # Refresh every 60 seconds ) ``` #### Manual Refresh Features are only refreshed when explicitly requested: ```elixir GrowthBook.init( client_key: "sdk-abc123", refresh_strategy: :manual # No automatic refresh ) # Later, manually trigger a refresh GrowthBook.FeatureRepository.refresh() ``` ### Supervision Tree Integration For production applications, add the `FeatureRepository` to your application's supervision tree: ```elixir # In your application.ex defmodule MyApp.Application do use Application require Logger def start(_type, _args) do children = [ # Your other supervised processes MyApp.Repo, MyAppWeb.Endpoint, # Add GrowthBook FeatureRepository {GrowthBook.FeatureRepository, client_key: System.get_env("GROWTHBOOK_CLIENT_KEY"), api_host: "https://cdn.growthbook.io", decryption_key: System.get_env("GROWTHBOOK_DECRYPTION_KEY"), swr_ttl_seconds: 60, refresh_strategy: :periodic, on_refresh: fn features -> Logger.info("GrowthBook features refreshed: #{map_size(features)}") end } ] opts = [strategy: :one_for_one, name: MyApp.Supervisor] case Supervisor.start_link(children, opts) do {:ok, pid} -> # Wait for features to load case GrowthBook.FeatureRepository.await_initialization( GrowthBook.FeatureRepository, 5000 ) do :ok -> Logger.info("GrowthBook features initialized successfully") {:error, reason} -> Logger.error("Failed to initialize GrowthBook features: #{inspect(reason)}") end {:ok, pid} error -> error end end end ``` ### Graceful Shutdown When your application shuts down, stop the FeatureRepository GenServer: ```elixir # Stops the FeatureRepository and releases resources GenServer.stop(GrowthBook.FeatureRepository) ``` ## Encryption & Security The Elixir SDK supports encrypted feature payloads and secure attribute hashing to protect sensitive data. ### Encrypted Features Encrypted features ensure that sensitive feature configurations are never exposed in plain text: ```elixir # Enable encryption by providing a decryption key GrowthBook.init( client_key: "sdk-abc123", api_host: "https://cdn.growthbook.io", decryption_key: System.get_env("GROWTHBOOK_DECRYPTION_KEY") ) ``` **Setup Steps:** 1. Enable "Encrypt SDK Payload" in your SDK Connection settings 2. Copy the encryption key from your SDK Connection 3. Store it securely in environment variables 4. Pass it to `GrowthBook.init/1` ### Secure Attributes When secure attribute hashing is enabled, you can safely target users based on sensitive attributes like email or phone numbers without exposing the actual values. #### Setup Enable secure attribute hashing in your SDK Connection, then hash sensitive attributes before passing them to GrowthBook: ```elixir defmodule MyApp.GrowthBook.SecureAttributes do @salt System.get_env("GROWTHBOOK_SECURE_ATTRIBUTE_SALT") def hash_attribute(value) when is_binary(value) do :crypto.hash(:sha256, value @salt) |> Base.encode16(case: :lower) end def build_secure_attributes(user) do %{ "id" => user.id, # Hash sensitive attributes "email" => hash_attribute(user.email), "phone" => hash_attribute(user.phone), # Non-sensitive attributes remain plain "country" => user.country, "plan" => user.plan } end end ``` #### Usage ```elixir # Build secure attributes for a user secure_attrs = MyApp.GrowthBook.SecureAttributes.build_secure_attributes(current_user) # Create context with hashed attributes context = GrowthBook.build_context(secure_attrs) # Features will use hashed attributes for targeting if GrowthBook.feature(context, "premium-feature").on? do show_premium_feature() end ``` ### Security Best Practices ```elixir # Store keys in environment variables config :my_app, :growthbook, client_key: System.get_env("GROWTHBOOK_CLIENT_KEY"), decryption_key: System.get_env("GROWTHBOOK_DECRYPTION_KEY"), secure_attribute_salt: System.get_env("GROWTHBOOK_SECURE_ATTRIBUTE_SALT") # Initialize with environment variables GrowthBook.init( client_key: Application.get_env(:my_app, :growthbook)[:client_key], decryption_key: Application.get_env(:my_app, :growthbook)[:decryption_key] ) ``` **Recommendations:** - **Never commit** encryption keys or salts to version control - **Use different keys** for each environment (dev, staging, production) - **Rotate keys regularly** and coordinate updates across all systems - **Monitor decryption failures** which may indicate key rotation issues - **Use secrets management** tools like Vault, AWS Secrets Manager, or similar ## Using Features Once GrowthBook is initialized, you can evaluate features and run experiments. ### Feature Evaluation ```elixir # Create context with user attributes context = GrowthBook.build_context(%{ "id" => "user-123", "email" => "user@example.com", "country" => "US", "plan" => "premium" }) # Boolean feature flag if GrowthBook.feature(context, "new-dashboard").on? do render_new_dashboard() else render_old_dashboard() end # Feature with value max_retries = GrowthBook.feature(context, "max-retries").value || 3 # Complex feature values config = GrowthBook.feature(context, "checkout-config").value timeout = config["timeout"] || 5000 enabled_methods = config["payment_methods"] || [] ``` ### Feature Result Properties The `GrowthBook.feature/2` function returns a result with these properties: ```elixir result = GrowthBook.feature(context, "my-feature") # Check if feature is enabled result.on? # true or false # Get the feature value result.value # Can be any JSON type # Check the source of the value result.source # :default_value, :force, :experiment, or :unknown_feature # Access experiment information (if feature is from an experiment) if result.source == :experiment do experiment = result.experiment experiment_result = result.experiment_result Logger.info("User in experiment: #{experiment.key}") end ``` ## Running Inline Experiments You can run experiments directly without defining them as features: ```elixir # Define an experiment experiment = %GrowthBook.Experiment{ key: "button-color-test", active?: true, coverage: 1.0, variations: ["red", "blue", "green"], weights: [0.5, 0.3, 0.2] } # Run the experiment result = GrowthBook.run(context, experiment) # Check if user is in the experiment if result.in_experiment? do color = result.value Logger.info("Assigned color: #{color}") set_button_color(color) else Logger.info("User not in experiment, using default") set_button_color("blue") end ``` ### Experiment Configuration The `Experiment` struct supports these properties: ```elixir %GrowthBook.Experiment{ # Required key: "my-experiment", variations: ["control", "treatment"], # Optional active?: true, # Whether experiment is active coverage: 1.0, # Percentage of users to include (0.0 to 1.0) weights: [0.5, 0.5], # Traffic distribution # Targeting condition: %{"country" => "US"}, # Targeting conditions hash_attribute: "id", # Attribute to use for hashing (default: "id") # Namespaces for experiment isolation namespace: {"pricing", 0, 0.5} # {name, start, end} } ``` ### Experiment Result The experiment result contains detailed information: ```elixir result = GrowthBook.run(context, experiment) result.in_experiment? # Boolean: whether user is in experiment result.variation_id # Integer: index of assigned variation result.value # The actual variation value result.hash_attribute # Attribute used for hashing result.hash_value # Value of the hash attribute ``` ## Tracking Experiments The SDK doesn't include built-in tracking callbacks, but you can implement tracking by checking experiment results: ```elixir # Run experiment result = GrowthBook.run(context, experiment) # Track if user is in experiment if result.in_experiment? do MyApp.Analytics.track_event("experiment_viewed", %{ experiment_id: experiment.key, variation_id: result.variation_id, user_id: context.attributes["id"] }) end # Use the variation case result.value do "control" -> render_control_version() "treatment" -> render_treatment_version() _ -> render_default_version() end ``` ## User Attributes Attributes are used for targeting and experiment assignment: ```elixir # Standard attributes context = GrowthBook.build_context(%{ "id" => "user-123", "email" => "user@example.com", "country" => "US", "browser" => "chrome" }) # Custom business attributes context = GrowthBook.build_context(%{ "id" => "user-123", "subscription_tier" => "premium", "lifetime_value" => 1500.00, "account_age_days" => 365, "is_high_value_customer" => true, "purchased_categories" => ["electronics", "books"], "feature_flags" => ["beta_access", "early_adopter"] }) ``` **Best Practices:** - Use consistent naming conventions (snake_case or camelCase) - Keep attribute values simple and serializable - Document custom attributes for your team - Consider attribute cardinality for targeting efficiency ## Context Struct The `GrowthBook.Context` struct is the core data structure: ```elixir %GrowthBook.Context{ enabled?: true, # Whether GrowthBook is enabled features: %{}, # Map of feature definitions attributes: %{}, # User attributes forced_variations: %{}, # Forced experiment variations for QA qa_mode?: false, # Disable randomization for testing url: nil # Optional URL for URL-based targeting } ``` You can create contexts manually or use the helper function: ```elixir # Using helper (recommended) context = GrowthBook.build_context(%{"id" => "user-123"}) # Manual creation (for advanced use cases) context = %GrowthBook.Context{ enabled?: true, features: my_features, attributes: %{"id" => "user-123"}, forced_variations: %{"experiment-1" => 1}, # Force variation for testing qa_mode?: false } ``` ## Error Handling The SDK is designed to fail gracefully: ```elixir # If initialization fails, you can still use manual features case GrowthBook.init(client_key: "invalid-key") do {:ok, :initialized} -> Logger.info("GrowthBook initialized") {:error, reason} -> Logger.warn("GrowthBook init failed: #{inspect(reason)}, using manual features") # Fall back to manual feature configuration fallback_features = load_fallback_features() end # Feature evaluation always returns a result result = GrowthBook.feature(context, "nonexistent-feature") # result.source will be :unknown_feature # result.value will be nil # result.on? will be false ``` ## Troubleshooting & Logging ### Common Issues #### Features Not Loading If features aren't loading, check the following: ```elixir # 1. Verify initialization succeeded case GrowthBook.init(client_key: "sdk-abc123") do {:ok, :initialized} -> Logger.info("GrowthBook initialized successfully") {:error, reason} -> Logger.error("Failed to initialize: #{inspect(reason)}") end # 2. Check if features are available case GrowthBook.FeatureRepository.await_initialization( GrowthBook.FeatureRepository, 5000 ) do :ok -> Logger.info("Features loaded") {:error, :timeout} -> Logger.error("Features not loaded within timeout") end # 3. Manually check feature repository state state = :sys.get_state(GrowthBook.FeatureRepository) Logger.info("Feature count: #{map_size(state.features)}") ``` #### Decryption Failures If encrypted features fail to decrypt: ```elixir # Ensure decryption key is set correctly decryption_key = System.get_env("GROWTHBOOK_DECRYPTION_KEY") if is_nil(decryption_key) or decryption_key == "" do Logger.error("GROWTHBOOK_DECRYPTION_KEY not set") end # Test decryption manually GrowthBook.init( client_key: "sdk-abc123", decryption_key: decryption_key, on_refresh: fn features -> Logger.info("Successfully decrypted #{map_size(features)} features") end ) ``` #### Network Issues For network connectivity problems: ```elixir # Test API connectivity api_host = "https://cdn.growthbook.io" client_key = System.get_env("GROWTHBOOK_CLIENT_KEY") case HTTPoison.get("#{api_host}/api/features/#{client_key}") do {:ok, %{status_code: 200}} -> Logger.info("API is reachable") {:ok, %{status_code: code}} -> Logger.error("API returned status code: #{code}") {:error, %{reason: reason}} -> Logger.error("Network error: #{inspect(reason)}") end ``` ### Logging Configuration Enable detailed logging to debug issues: ```elixir # config/config.exs config :logger, :console, format: "$time $metadata[$level] $message\n", metadata: [:request_id, :module, :function] # Set log level for GrowthBook config :logger, level: :debug # In your application require Logger GrowthBook.init( client_key: "sdk-abc123", on_refresh: fn features -> Logger.info("GrowthBook: Features refreshed", feature_count: map_size(features), timestamp: DateTime.utc_now() ) end ) ``` ### Debug Helper Module Create a helper module for debugging GrowthBook: ```elixir defmodule MyApp.GrowthBookDebug do require Logger def inspect_context(context) do Logger.debug(""" GrowthBook Context: Enabled: #{context.enabled?} Features: #{map_size(context.features)} Attributes: #{inspect(context.attributes)} QA Mode: #{context.qa_mode?} """) end def inspect_feature_result(key, result) do Logger.debug(""" Feature: #{key} Value: #{inspect(result.value)} On: #{result.on?} Source: #{result.source} In Experiment: #{result.source == :experiment} """) end def list_all_features do state = :sys.get_state(GrowthBook.FeatureRepository) state.features |> Map.keys() |> Enum.each(fn key -> Logger.info("Feature: #{key}") end) end end # Usage context = GrowthBook.build_context(%{"id" => "user-123"}) MyApp.GrowthBookDebug.inspect_context(context) result = GrowthBook.feature(context, "my-feature") MyApp.GrowthBookDebug.inspect_feature_result("my-feature", result) ``` ### Health Checks Implement health checks for monitoring: ```elixir defmodule MyApp.HealthCheck do def growthbook_status do try do case Process.whereis(GrowthBook.FeatureRepository) do nil -> {:error, "FeatureRepository not running"} pid when is_pid(pid) -> state = :sys.get_state(pid) feature_count = map_size(state.features) cond do feature_count == 0 -> {:warning, "No features loaded"} true -> {:ok, "#{feature_count} features loaded"} end end rescue e -> {:error, "Health check failed: #{inspect(e)}"} end end end # Use in a Phoenix health endpoint defmodule MyAppWeb.HealthController do use MyAppWeb, :controller def show(conn, _params) do growthbook_status = MyApp.HealthCheck.growthbook_status() status = case growthbook_status do {:ok, _} -> :ok {:warning, _} -> :degraded {:error, _} -> :error end json(conn, %{ status: status, growthbook: growthbook_status }) end end ``` ## Integrations The Elixir SDK integrates seamlessly with popular Elixir frameworks and libraries. ### Phoenix Web Application Integrate GrowthBook with Phoenix for feature flags in your web application: ```elixir # lib/my_app_web/plugs/growthbook_plug.ex defmodule MyAppWeb.GrowthBookPlug do import Plug.Conn def init(opts), do: opts def call(conn, _opts) do # Build user attributes from session/assigns user = conn.assigns[:current_user] attributes = %{ "id" => user_id(user), "email" => user && user.email, "country" => get_country_from_ip(conn.remote_ip), "user_agent" => get_req_header(conn, "user-agent") |> List.first(), "url" => conn.request_path, "plan" => user && user.subscription_plan } # Create GrowthBook context context = GrowthBook.build_context(attributes) # Store context in conn.assigns for use in controllers/views assign(conn, :growthbook, context) end defp user_id(nil), do: nil defp user_id(user), do: to_string(user.id) defp get_country_from_ip(_ip) do # Implement IP geolocation "US" end end # lib/my_app_web/router.ex defmodule MyAppWeb.Router do use MyAppWeb, :router pipeline :browser do plug :accepts, ["html"] plug :fetch_session plug :fetch_live_flash plug :put_root_layout, {MyAppWeb.LayoutView, :root} plug :protect_from_forgery plug :put_secure_browser_headers plug :fetch_current_user plug MyAppWeb.GrowthBookPlug # Add GrowthBook plug end # Your routes... end # lib/my_app_web/controllers/dashboard_controller.ex defmodule MyAppWeb.DashboardController do use MyAppWeb, :controller def index(conn, _params) do # Access GrowthBook context from conn.assigns gb = conn.assigns.growthbook # Use feature flags to control UI show_new_dashboard = GrowthBook.feature(gb, "new-dashboard").on? max_items = GrowthBook.feature(gb, "dashboard-max-items").value || 10 # Track experiment if user is in one color_result = GrowthBook.feature(gb, "dashboard-theme-color") if color_result.source == :experiment do track_experiment(conn, color_result.experiment, color_result.experiment_result) end render(conn, "index.html", new_dashboard: show_new_dashboard, max_items: max_items, theme_color: color_result.value ) end defp track_experiment(conn, experiment, result) do # Track to your analytics service MyApp.Analytics.track(conn.assigns.current_user, "experiment_viewed", %{ experiment_id: experiment.key, variation_id: result.variation_id }) end end ``` ### Phoenix LiveView Use GrowthBook with Phoenix LiveView for real-time feature flag updates: ```elixir # lib/my_app_web/live/dashboard_live.ex defmodule MyAppWeb.DashboardLive do use MyAppWeb, :live_view require Logger @impl true def mount(_params, session, socket) do # Subscribe to feature updates if desired if connected?(socket) do Phoenix.PubSub.subscribe(MyApp.PubSub, "growthbook:features") end # Get user from session user = get_user_from_session(session) # Build GrowthBook context gb_context = GrowthBook.build_context(%{ "id" => user.id, "email" => user.email, "plan" => user.subscription_plan, "country" => user.country }) socket = socket |> assign(:user, user) |> assign(:growthbook, gb_context) |> assign_features() {:ok, socket} end @impl true def handle_info({:features_updated, _features}, socket) do # Rebuild context with updated features gb_context = GrowthBook.build_context(socket.assigns.user.attributes) socket = socket |> assign(:growthbook, gb_context) |> assign_features() |> put_flash(:info, "Features updated") {:noreply, socket} end defp assign_features(socket) do gb = socket.assigns.growthbook socket |> assign(:new_dashboard, GrowthBook.feature(gb, "new-dashboard").on?) |> assign(:max_items, GrowthBook.feature(gb, "max-items").value || 20) |> assign(:theme, GrowthBook.feature(gb, "dashboard-theme").value || "light") end @impl true def render(assigns) do ~H""" """ end defp get_user_from_session(session) do # Get user from session # Implementation depends on your auth system end end # Set up PubSub notifications when features refresh # In your GrowthBook initialization (application.ex) GrowthBook.init( client_key: System.get_env("GROWTHBOOK_CLIENT_KEY"), on_refresh: fn features -> Phoenix.PubSub.broadcast( MyApp.PubSub, "growthbook:features", {:features_updated, features} ) end ) ``` ### View Helpers Create view helpers for easy feature flag usage in templates: ```elixir # lib/my_app_web/views/growthbook_helpers.ex defmodule MyAppWeb.GrowthBookHelpers do def feature_on?(conn, feature_key) do conn.assigns.growthbook |> GrowthBook.feature(feature_key) |> Map.get(:on?) end def feature_value(conn, feature_key, default \\ nil) do conn.assigns.growthbook |> GrowthBook.feature(feature_key) |> Map.get(:value) |> case do nil -> default value -> value end end end # In your templates ``` ## Supported Features --- ## Flutter SDK import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Flutter This SDK supports the following versions: - **Android version 21 & above** - **iOS version 12 & Above** - **Apple TvOS version 13 & Above** - **Apple WatchOS version 7 & Above** ## Installation Add this to your `pubspec.yaml` file ```yaml growthbook_sdk_flutter: ^4.0.0 ``` ## Quick Usage Create a `GrowthBookSDK` instance with `GBSDKBuilderApp`, set attributes, then evaluate features or run experiments. ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", // Client key from GrowthBook hostURL: "https://cdn.growthbook.io", // or your Proxy URL attributes: { "id": "123", "env": "dev", "betaUser": true, }, growthBookTrackingCallBack: (exp, result) { // Track exposures }, ).initialize(); final feature = sdk.feature("my-feature"); if (feature.on) { // Feature is enabled } ``` ## Using Features The `feature` method takes a String feature name and returns a `GBFeatureResult` object with a few useful properties: - **value** (`dynamic`) - The assigned value of the feature - **on** (`bool`) - The value cast to a boolean - **off** (`bool`) - The value cast to a boolean and then negated - **source** (`String`) - Why the value was assigned to the user. One of "unknownFeature", "defaultValue", "force", or "experiment" When the source is "experiment", there are 2 additional properties that tell you which experiment was used and more details about the result of the experiment: - **experiment** (`GBExperiment`) - **experimentResult** (`GBExperimentResult`) Here are some examples: ```dart GBFeatureResult feature = gb.feature("my-feature") // Do something if feature is truthy if (feature.on) { } // Do something if feature is falsy if (feature.off) { } // Print the actual value of the feature // (depending on the feature, might be a string, number, boolean, etc.) print(feature.value) // Print the experiment id used to assign the feature value if (feature.source == "experiment") { Println(feature.experiment.key) } ``` ## Attributes Attributes define the current user and request "context" used for targeting rules and bucketing. ### Common attributes - `id` (string): primary user identifier for consistent bucketing - `deviceId` (string): device/install-scoped fallback when user not logged in - `country`, `locale` (string) - `plan`, `company`, `role` (string) - `loggedIn` (bool) - `appVersion` (string, supports semver targeting) Use `setAttributes()` method to set the user context. ```dart sdk.setAttributes({ "id": "123", "country": "US", "plan": "pro", "appVersion": "1.3.0", }); ``` ### What is the user context? - Identity: stable identifiers for bucketing (e.g., `id`, and optionally a logged-out fallback like `deviceId`). - Demographics and traits: e.g., `country`, `company`, `plan`, `loggedIn`. - App/runtime context: e.g., `appVersion`, `platform`, `locale`. - Request context (if applicable): e.g., `url`, `path`, `device`. Data types supported include strings, numbers, booleans, and arrays/objects for JSON-based conditions. ### Identity changes (login/logout) - On login, switch from `deviceId` to `id` (stable user id). - On logout, remove `id` and use `deviceId` until the next login. ```dart // Logged-out (device-scoped) sdk.setAttributes({ "deviceId": "device-abc", "loggedIn": false, }); // After login (user-scoped) sdk.setAttributes({ "id": "user-123", "loggedIn": true, }); ``` If using Remote Evaluation, consider limiting network calls to meaningful identity changes with `cacheKeyAttributes` and refreshing when identity changes: ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://gb-proxy.example.com", remoteEval: true, cacheKeyAttributes: ["id", "email"], attributes: {"deviceId": "device-abc"}, ).initialize(); // Later, after login sdk.setAttributes({ "id": "user-123", "email": "user@example.com", }); await sdk.refreshForRemoteEval(); ``` ## Experimentation (A/B Testing) To run the experiements with `GrowthBook`, use `run` method, which takes a `GBExperiment` object as an argument and returns a `GBExperimentResult` object: ```dart var exp = GBExperiment() exp.key = "my-experiment" exp.variations = List.of("control", "variation") var result = gb.run(exp) // Either "control" or "variation" print(result.value) ``` The `GBExperiment` class has two required properties - `key` and `variations`. There are also a number of optional properties: - **key** (`String`) - The unique identifier for this experiment - **variations** (`dynamic[]`) - Array of variations to decide between - **weights** (`double[]`) - How to weight traffic between variations. Must add to 1. - **active** (`bool`) - If set to false, always return the control (first variation) - **coverage** (`double`) - What percent of users should be included in the experiment (between 0 and 1, inclusive) - **condition** (`GBCondition`) - Optional targeting condition - **namespace** (`[String, int, int]`) - Adds the experiment to a namespace - **force** (`int`) - All users included in the experiment will be forced into the specific variation index - **hashAttribute** (`String`) - What user attribute should be used to assign variations (defaults to `id`) The `GBExperimentResult` object returns the following properties: - **inExperiment** (`bool`) - **variationId** (`int`) - The array index of the assigned variation - **value** (`dynamic`) - The value of the assigned variation - **hashAttribute** (`String`) - The user attribute used to assign a variation - **hashValue** (`String`) - The value of the attribute used to assign a variation ### Tracking & Subscriptions Use `growthBookTrackingCallBack` to receive experiment exposure events whenever a user is assigned to a variation (via features or inline experiments). For feature usage events (non-experiment), wrap your `feature(...)` calls in a small utility to emit custom app analytics. ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://cdn.growthbook.io", growthBookTrackingCallBack: (exp, result) { // e.g., send to analytics }, ).initialize(); ``` To Subscribe : ```dart final exposures = StreamController>.broadcast(); final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://cdn.growthbook.io", growthBookTrackingCallBack: (exp, result) { exposures.add({ "experimentKey": exp.key, "variationId": result.variationId, "featureId": result.featureId, }); }, ).initialize(); final sub = exposures.stream.listen((e) { // send to analytics }); // Later, unsubscribe to avoid leaks await sub.cancel(); ``` :::info When subscriptions fire vs. don't fire When subscriptions fire vs. don't fire - Fire: on first assignment per experiment key and after assignments change due to attribute updates or payload changes. - Don't fire: repeated reads of the same assignment that hasn't changed. ::: #### Efficient subscription patterns - Keep callbacks lightweight; offload heavy work to background tasks. - Coalesce multiple exposure events before sending to analytics to reduce overhead. - Include `experiment.key`, `variationId`, and optional `featureId` for correlation. #### Unsubscribing and memory management - Remove any of your app-layer listeners in `dispose()`. - Dispose the SDK instance when not needed (e.g., app shutdown) to release resources and close streams. ## Loading Features #### Built-in fetching and caching If you pass a `hostURL` and `apiKey` into the builder, the SDK handles network requests, caching, retry/backoff, and decryption (when configured in your SDK connection). You can enable streaming updates with `backgroundSync: true`. ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://cdn.growthbook.io", backgroundSync: false, ).initialize(); // Manually refresh (e.g., on app start or navigation) await sdk.refreshCache(); ``` #### Custom integration (local evaluation) If you prefer to control network and caching yourself, you can set a payload directly on the SDK. This enables fully local evaluation and offline-first behavior. ```dart await sdk.setPayload({ "features": { "feature-1": {"defaultValue": true}, "feature-2": {"defaultValue": "blue"} } }); ``` ## Caching The SDK persists downloaded feature payloads and related metadata. Configure TTL to control how long cached features are considered fresh before a background refresh. - To use Stale-While-Revalidate strategy with TTL and streaming: ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://cdn.growthbook.io", ttlSeconds: 300, backgroundSync: true, // keep a live stream for fast updates ).initialize(); ``` - Manual refresh at lifecycle boundaries: ```dart // App resume, significant navigation, pull-to-refresh, etc. await sdk.refreshCache(); ``` - Identity or environment switch: ```dart // Switch attributes to new identity, then refresh sdk.setAttributes({"id": "new-user"}); await sdk.refreshCache(); // If you store sticky assignments yourself, clear the old user's keys await myStickyStorage.clearForUser("old-user"); ``` #### Web-specific caching considerations - For Flutter Web, ensure your CDN respects cache-control headers for GrowthBook endpoints. - Avoid overly aggressive service worker caching for evaluated payloads unless you manage invalidation carefully. ## Real-time Updates (SSE) ### Setup and lifecycle - Initialization - Enable `backgroundSync: true` when building the SDK instance. - Optionally pass `streamingRequestHeaders` (e.g., auth) and `Last-Event-Id` to resume after restarts. - Refresh handler - Use `onFeaturesRefreshed` to update UI or invalidate caches after new data is applied. - The callback fires for both streaming updates and manual refreshes. - Teardown - Dispose your SDK instance when your app shuts down. The connection will be closed automatically. ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://cdn.growthbook.io", backgroundSync: true, onFeaturesRefreshed: (success) { if (success) { // e.g., notify listeners / rebuild widgets } else { // network failure or payload issue } }, ).initialize(); ``` You can also add custom headers for secure requests: ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://cdn.growthbook.io", requestHeaders: {"Authorization": "Bearer "}, streamingRequestHeaders: { "Authorization": "Bearer ", // Optionally resume after disconnects "Last-Event-Id": "" }, backgroundSync: true, ).initialize(); ``` ### Network failures and reconnection - The SDK uses an exponential backoff strategy on streaming errors and attempts to auto-reconnect. - Provide stable connectivity hints to users only when necessary; otherwise allow background reconnection. - Persist and reuse the `Last-Event-Id` to avoid duplicate events on resume. ```dart // Example: Persist last event id you receive from your stream handler await storage.write(key: 'gb_last_event_id', value: lastEventId); // Then pass it back during init via `streamingRequestHeaders` as shown above ``` #### Performance Considerations - Prefer streaming when you need near real-time flag updates (admin toggles, ops tooling). - Prefer polling/manual refresh when updates are infrequent or the app is latency- or battery-sensitive. - Background sync opens a single lightweight SSE connection; avoid running multiple SDK instances with streaming in the same app. ## Remote Evaluation :::tip See the [Remote Evaluation overview](/self-host/remote-evaluation) for more information about what Remote Evaluation is, how it works, and deployment options. ::: Run GrowthBook in Remote Evaluation mode to evaluate flags on a private server (e.g., GrowthBook Proxy). Sensitive rules never reach the client. :::note Sticky Bucketing with Remote Evaluation If you want Sticky Bucketing with Remote Evaluation, configure your remote backend (e.g., GrowthBook Proxy) with a persistent store (e.g., Redis). You do not need to provide a sticky service on the client. ::: ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://gb-proxy.yourcompany.com", remoteEval: true, // Optional: only trigger a new evaluation when selected attributes change cacheKeyAttributes: ["id", "email"], attributes: {"id": "123", "email": "user@example.com"}, ).initialize(); // Manually trigger a remote evaluation when attributes change sdk.setAttributes({"plan": "pro"}); await sdk.refreshForRemoteEval(); ``` ### Remote vs local evaluation - Use remote evaluation when: - You need to keep targeting rules and unused variations off-device for privacy/compliance. - Payload size is large and you want device-optimized responses for the current user. - Use local evaluation when: - You need offline-first behavior or minimal round trips. - The full feature payload is small and regularly reused. #### Configuration - Endpoint: Set `hostURL` to your GrowthBook Proxy or secured backend base URL. - Auth: Send authentication via `requestHeaders` to both fetching and remote eval endpoints. - Cache keys: Use `cacheKeyAttributes` to limit remote calls to only when relevant identity fields change. ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://gb-proxy.example.com", remoteEval: true, cacheKeyAttributes: ["id", "email"], requestHeaders: {"Authorization": "Bearer "}, attributes: {"id": "u_123", "email": "u@example.com"}, ).initialize(); ``` ### Debugging Remote Evaluation - Verify required attributes are present before calling `refreshForRemoteEval()`. - Log the response status/source from your SDK callbacks to confirm whether updates came from network or cache. - Check your proxy/server logs for rejected requests, auth failures, or schema mismatches. #### Security considerations - Treat the `apiKey` as public. Protect sensitive data with server-side auth on your proxy. - Use short-lived tokens in `requestHeaders` when possible and rotate regularly. - Avoid putting PII directly in attributes unless hashed/secured per your policies. #### Fallback strategies - If remote eval fails, keep using the last known values from cache and retry later. - Consider a hybrid approach: bootstrap with a local payload and switch to remote eval when online. - Implement a user-visible retry or manual refresh action when business-critical. ## Sticky Bucketing Sticky bucketing ensures users see the same experiment variant across sessions. Read more about [Sticky Bucketing Feature](https://docs.growthbook.io/app/sticky-bucketing) - Enable Sticky Bucketing on the experiment or feature rule in GrowthBook. - Provide a stable identity in attributes (e.g., `id`) after login; use a device-scoped fallback when logged out. - Use a globally unique `id` for logged-in users; avoid mutable identifiers. - For logged-out sessions, use a device/install identifier and switch to `id` on login. - When identity changes (login/logout), update attributes immediately to avoid cross-user assignment mixing. Implement a `GBStickyBucketService` to persist assignments. Attach your `GBStickyBucketService` when building the SDK to persist assignments across sessions. ```dart class MyAppStickyBucketService extends GBStickyBucketService { @override Future?> getAllAssignments( Map attributes, ) async { // Retrieve from local storage // ... return null; } @override Future saveAssignments( Map attributes, Map assignments, ) async { // Save to local storage // ... } } final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", stickyBucketService: MyAppStickyBucketService(), ).initialize(); ``` For Remote evaluations, if your Remote Eval backend (e.g., GrowthBook Proxy) is configured for sticky bucketing, persistence happens server-side and you do not need to provide a client service. ```dart // Remote evaluation with server-side sticky (no client sticky service needed) final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://gb-proxy.yourcompany.com", remoteEval: true, attributes: {"id": "user_123"}, ).initialize(); ``` ### Custom Sticky Bucketing Service configurations - Namespacing: use an environment-specific key prefix (e.g., `prod_`, `staging_`). - Storage backend: choose durable storage for your platform (`SharedPreferences`, secure storage, SQLite). - Migration: when changing storage layout, migrate existing assignment keys to preserve continuity. ### Troubleshooting stale assignments - Logout or user-switch: clear assignments for the previous identity or use identity-specific keys. - Experiment retired/changed: allow non-sticky behavior to take effect; optionally prune retired experiment keys. - Unexpected variant: confirm which identity key is being used and verify read/write paths in your storage. See the Flutter repo for examples and updates: [growthbook-flutter](https://github.com/growthbook/growthbook-flutter). ## Encrypted features Configure SDK Connection to deliver encrypted features, provide the decryption key during initialization. ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://cdn.growthbook.io", decryptionKey: "", ).initialize(); ``` Or prefer using a Remote Evaluation backend (e.g., GrowthBook Proxy) that decrypts server-side. This keeps secrets off the device. ```dart final sdk = await GBSDKBuilderApp( apiKey: "sdk-abc123", hostURL: "https://gb-proxy.yourcompany.com", // Proxy handles decryption remoteEval: true, attributes: {"id": "user_123"}, ).initialize(); ``` ### Performance optimizations - Track refresh times and payload sizes; reduce attribute surface area where possible. - Avoid multiple SDK instances with streaming enabled; prefer a single shared instance. - Use `onFeaturesRefreshed` to scope UI rebuilds and avoid unnecessary widget tree updates. #### Memory usage patterns - Keep attribute maps small and reuse them when doing partial updates. - Periodically prune stale sticky assignments if you implement a custom store. - Avoid holding multiple SDK instances; share a single instance across the app. ## Error handling and debugging ### Debug logging configuration ```dart // Simple logger wrapper for development void gbLog(String message) { assert(() { // only in debug mode // ignore: avoid_print print('[GrowthBook] ' + message); return true; }()); } // Example usage gbLog('Initializing GrowthBook'); ``` ### Common error scenarios and solutions - Initialization returned from cache only - Ensure network connectivity and correct `hostURL`/`apiKey`. - Call `await sdk.refreshCache()` on resume or via pull-to-refresh. - Unauthorized (401/403) - Verify `requestHeaders` (e.g., Bearer token) and that your proxy/CDN forwards headers. - Decryption errors - Confirm `decryptionKey` matches the SDK Connection and is valid; prefer server-side decryption. #### Feature evaluation debugging techniques ```dart final res = sdk.feature('checkout_v2'); gbLog('feature=checkout_v2 value=${res.value} source=${res.source}'); if (res.source == 'experiment') { gbLog('exp=' + (res.experiment?.key ?? 'n/a') + ' varId=${res.experimentResult?.variationId}'); } ``` Inspect attributes used during evaluation: ```dart // Keep your own copy of the attributes you set final currentAttrs = {"id": "123", "country": "US"}; sdk.setAttributes(currentAttrs); gbLog('attrs=' + currentAttrs.toString()); ``` ### Network connectivity troubleshooting ```dart try { await sdk.refreshCache(); gbLog('Refresh succeeded'); } catch (e) { gbLog('Refresh error: ' + e.toString()); } ``` If streaming is enabled, rely on auto-reconnect with backoff. Allow a manual refresh UI if business-critical. ## Platform-specific considerations ### Web vs mobile behavior differences - SSE support and networking may differ in browsers; validate CORS and service worker behavior for Flutter Web. - Storage persistence differs (IndexedDB/localStorage on Web vs SharedPreferences/Secure Storage on mobile). - For Web, prefer IndexedDB for larger storage; be careful with quota and private browsing modes. - For Mobile: use `SharedPreferences` or secure storage for sticky assignments; consider SQLite for large datasets. ### iOS/Android specific considerations - iOS backgrounding may pause network activity; prefer manual refresh on foreground via app lifecycle hooks. - Android Doze/App Standby can throttle background sync; design UX that tolerates delayed updates. Do not rely on continuous streaming while the app is fully backgrounded; on resume, call `refreshCache()` and rely on backoff reconnects. See Flutter SDK repo: `https://github.com/growthbook/growthbook-flutter` ## Supported Features --- ## Go SDK import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # GrowthBook Go SDK ## Requirements - Go version 1.21 or higher (tested with 1.21, 1.22, and 1.23) ## Installation ``` go get github.com/growthbook/growthbook-golang ``` ## Quick Usage ```go import ( "context" "log" gb "github.com/growthbook/growthbook-golang" ) // Create a new client instance with a client key and a data source that loads features // in the background using an SSE stream. Pass the client's options to the NewClient function. client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-XXXX"), gb.WithSseDataSource(), ) defer client.Close() if err != nil { log.Fatal("Client initialization failed: ", err) } // The data source starts asynchronously. Use EnsureLoaded to wait until the client data // is initialized for the first time. if err := client.EnsureLoaded(context.Background()); err != nil { log.Fatal("Data loading failed: ", err) } // Create a child client with specific attributes. attrs := gb.Attributes{"id": 100, "user": "user1"} child, err := client.WithAttributes(attrs) if err != nil { log.Fatal("Child client creation failed: ", err) } // Evaluate a text feature. buttonColor := child.EvalFeature(context.Background(), "buy-button-color") if buttonColor.Value == "blue" { // Perform actions for blue button. } // Evaluate a boolean feature. darkMode := child.EvalFeature(context.Background(), "dark-mode") if darkMode.On { // Enable dark mode. } ``` ### Client The client is the core component of the GrowthBook SDK. After installing and importing the SDK, create a single shared instance of `growthbook.Client` using the `growthbook.NewClient` function with a list of options. You can customize the client with options such as a custom logger, client key, decryption key, default attributes, or a feature list loaded from JSON. The client is thread-safe and can be safely used from multiple goroutines. While you can evaluate features directly using the main client instance, it's recommended to create child client instances that include session- or query-specific data. To create a child client with local attributes, call `client.WithAttributes`: ```go attrs := gb.Attributes{"id": 100, "user": "Bob"} child, err := client.WithAttributes(attrs) ``` You can then evaluate features using the child client: ```go res := child.EvalFeature(context.Background(), "main-button-color") ``` Additional options, such as `WithLogger`, `WithUrl`, and `WithAttributesOverrides`, can also be used to customize child clients. Since child clients share data with the main client instance, they will automatically receive feature updates. To stop background updates, call `client.Close()` on the main client instance when it is no longer needed. #### Additional options for sticky bucketing: - `WithStickyBucketService`: Provides a service implementation for storing and retrieving sticky bucket assignments - `WithStickyBucketAttributes`: Sets specific attributes to use for sticky bucketing (if different from regular attributes) ```go // Configure a client with sticky bucketing client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-XXXX"), gb.WithStickyBucketService(NewInMemoryStickyBucketService()), ) ``` --- ## Using Features The primary method, `client.EvalFeature(ctx, key)`, accepts a feature key and uses the stored feature definitions and attributes to evaluate the feature value. It returns a `FeatureResult` value that includes detailed information about why the value was assigned to the user: - `Value`: The JSON value of the feature (or `nil` if not defined), represented as a `FeatureValue` (an alias for `interface{}`, using Go's default behavior for JSON). - `On` and `Off`: The JSON value cast as booleans (to make your code easier to read). - `Source`: A value of type `FeatureResultSource` that explains why the value was assigned to the user. Possible values include `UnknownFeatureResultSource`, `DefaultValueResultSource`, `ForceResultSource`, or `ExperimentResultSource`. - `Experiment`: Information about the experiment (if any) used to assign the value. - `ExperimentResult`: The result of the experiment (if any) that determined the value. Here's an example that uses all of these fields: ```go result, err := client.EvalFeature(context.TODO(), "my-feature") if err != nil { // Handle the error } // The JSON value (which may be null, a string, boolean, number, array, or object). fmt.Println(result.Value) if result.On { // The feature value is truthy (in a JavaScript sense). } if result.Off { // The feature value is falsy. } // If the feature value was assigned as part of an experiment: if result.Source == gb.ExperimentResultSource { // Get all the possible variations that could have been assigned. fmt.Println(result.Experiment.Variations) } ``` ## Loading Features and Experiments For the GrowthBook SDK to function, it requires feature and experiment definitions from the GrowthBook API. There are several ways to provide this data to the SDK. ### Automatic Features Refresh The Go SDK provides multiple mechanisms for automatically keeping your feature definitions up to date. #### Server-Sent Events (SSE) SSE provides real-time feature updates with minimal overhead. When you use `WithSseDataSource()`, the SDK establishes a persistent connection to receive live updates whenever features change. ```go // Create a client with SSE for real-time updates client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithSseDataSource(), // Enable SSE streaming ) if err != nil { log.Fatal("Failed to create client:", err) } defer client.Close() // Wait for initial feature load if err := client.EnsureLoaded(context.Background()); err != nil { log.Fatal("Failed to load features:", err) } // Features will now update automatically via SSE ``` **Benefits of SSE:** - Real-time updates without polling overhead - Lower latency for feature changes - Reduced server load - Automatic reconnection on connection loss #### Polling Data Source For environments where SSE isn't supported, use polling to periodically fetch feature updates: ```go import "time" // Poll for updates every 30 seconds client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithPollDatasource(30 * time.Second), // Poll every 30 seconds ) if err != nil { log.Fatal("Failed to create client:", err) } defer client.Close() // Wait for initial feature load if err := client.EnsureLoaded(context.Background()); err != nil { log.Fatal("Failed to load features:", err) } ``` **When to Use Polling:** - SSE is blocked by firewalls or proxies - Running in restricted network environments - Need predictable update intervals - Simpler infrastructure requirements #### Choosing Between SSE and Polling | Feature | SSE | Polling | | ------------------------ | -------------------------------- | ----------------------------------- | | **Latency** | Real-time (milliseconds) | Based on interval (seconds/minutes) | | **Server Load** | Low (one connection) | Higher (periodic requests) | | **Network Requirements** | Persistent connection | Intermittent requests | | **Use Case** | Production apps, real-time needs | Restricted networks, batch updates | ### Built-in Fetching and Caching The loading of features is an asynchronous process so that your app is not blocked while waiting, and it can continue its initialization. If you need to ensure that the definitions are loaded, use the `client.EnsureLoaded` call. This will block until the loading process finishes and will return an error if any failures occur. ```go // Create client with automatic updates client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithSseDataSource(), ) if err != nil { log.Fatal(err) } // EnsureLoaded blocks until features are ready ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() if err := client.EnsureLoaded(ctx); err != nil { log.Fatal("Features not loaded in time:", err) } // Features are now loaded and will update automatically fmt.Println("Features loaded successfully") ``` **Key Points:** - `EnsureLoaded` is thread-safe and can be called from multiple goroutines simultaneously - Both `NewClient` and `EnsureLoaded` respect the contexts passed to them - The features cache is shared among all child client instances created via `client.WithXXX` calls - Use context timeouts to prevent indefinite blocking ### Custom Integration Feature definitions are stored in the client's shared data. Normally, the data source will download them from the GrowthBook site, but you can provide an initial set during the `NewClient` call using the `WithFeatures`, `WithJsonFeatures`, or `WithEncryptedJsonFeature` options. ```go // Load features from a local JSON file featuresJSON, err := os.ReadFile("features.json") if err != nil { log.Fatal(err) } client, err := gb.NewClient( context.Background(), gb.WithJsonFeatures(featuresJSON), ) if err != nil { log.Fatal(err) } ``` It is also possible to update the shared feature definitions using the `SetXXXFeatures` client methods: ```go // Update features dynamically newFeaturesJSON := []byte(`{"features": {...}}`) if err := client.UpdateFromApiResponseJSON(newFeaturesJSON); err != nil { log.Error("Failed to update features:", err) } ``` ### Manual Feature Updates You can manually trigger feature updates when needed: ```go // Force a feature refresh client.RefreshFeatures(context.Background()) // Or update from a JSON response apiResponse, err := fetchFeaturesFromAPI() if err != nil { log.Error("Failed to fetch features:", err) return } if err := client.UpdateFromApiResponseJSON(apiResponse); err != nil { log.Error("Failed to update features:", err) } ``` ## Attributes You can specify attributes about the current user and request. These attributes are used for the following purposes: - Feature targeting (for example, paid users receive one value, while free users receive another). - Assigning persistent variations in A/B tests (for example, a user with id "123" always gets variation B). Attributes can be any JSON data type—boolean, float, string, array, or object—and are represented by the `Attributes` type, which is an alias for Go's generic `map[string]interface{}` type used for JSON objects. If you know the attributes upfront, you can pass them into `NewClient` using the `WithAttributes` option: ```go attrs := gb.Attributes{ "id": "123", "loggedIn": true, "deviceId": "abc123def456", "company": "acme", "paid": false, "url": "/pricing", "browser": "chrome", "mobile": false, "country": "US", } client, err := gb.NewClient(context.Background(), gb.WithAttributes(attrs), ) ``` You can also create a child client instance with updated attributes using the `WithAttributes` method: ```go attrs := gb.Attributes{ "id": "100", "mobile": true, } client, err := client.WithAttributes(attrs) ``` This will completely overwrite the existing attributes object with the values you provide. If you want to merge the new attributes with the existing ones instead, you can use the `WithAttributeOverrides` method: ```go client, err := client.WithAttributeOverrides(attrs) ``` This method updates only the fields provided in `attrs`, keeping the other fields from the original client instance. Be aware that changing attributes may change the assigned feature values. This can be disorienting to users if not handled carefully. A common approach is to refresh attributes only on navigation, when the window is focused, or after a user performs a major action such as logging in. ### Secure Attributes When _secure attribute hashing_ is enabled, all targeting conditions in the SDK payload referencing attributes with datatype `secureString` or `secureString[]` will be anonymized via SHA-256 hashing. This allows you to safely target users based on sensitive attributes. You must enable this feature in your SDK Connection for it to take effect. If your SDK Connection has secure attribute hashing enabled, you will need to manually hash any `secureString` or `secureString[]` attributes that you pass into the GrowthBook SDK. To hash an attribute, use Go's `crypto/sha256` package to compute the SHA-256 hashed value of your attribute _plus_ your organization's secure attribute salt. ```go import ( "crypto/sha256" "encoding/hex" gb "github.com/growthbook/growthbook-golang" ) // Your secure attribute salt (set in Organization Settings) salt := "f09jq3fij" // Hashing a secureString attribute userEmail := user.Email hasher := sha256.New() hasher.Write([]byte(salt + userEmail)) hashedEmail := hex.EncodeToString(hasher.Sum(nil)) // Hashing a secureString[] attribute userTags := user.Tags hashedTags := make([]string, len(userTags)) for i, tag := range userTags { hasher := sha256.New() hasher.Write([]byte(salt + tag)) hashedTags[i] = hex.EncodeToString(hasher.Sum(nil)) } // Create attributes with hashed values attrs := gb.Attributes{ "id": user.ID, "loggedIn": true, "email": hashedEmail, "tags": hashedTags, } // Create a client with the hashed attributes client, err := client.WithAttributes(attrs) if err != nil { // Handle error } ``` ### Custom Attributes You can define custom attributes for advanced targeting scenarios beyond standard user properties: ```go // Standard and custom business attributes attrs := gb.Attributes{ // Standard attributes "id": user.ID, "email": user.Email, "country": user.Country, // Custom business attributes "lifetimeValue": user.LifetimeValue, "subscriptionTier": user.Tier, "accountAge": user.AccountAge, "lastPurchaseDate": user.LastPurchase.Unix(), // Computed attributes "isHighValueCustomer": user.LifetimeValue > 1000, "isPremium": user.Tier == "premium" || user.Tier == "enterprise", // Array attributes for complex targeting "purchasedCategories": user.Categories, "enabledFeatures": user.Features, // Request-specific attributes "requestPath": req.URL.Path, "userAgent": req.UserAgent(), "clientIP": req.RemoteAddr, } client, err := client.WithAttributes(attrs) if err != nil { log.Error("Failed to set attributes:", err) } ``` **Best Practices for Custom Attributes:** - Use consistent naming conventions across your application - Keep attribute values serializable (avoid complex nested structures) - Consider attribute cardinality for targeting rules - Document custom attributes in your team's documentation ## Encrypted Features The Go SDK supports encrypted feature payloads to protect sensitive feature configurations from being exposed in transit or at rest. #### Setup Encrypted Features 1. **Enable encryption** in your GrowthBook SDK Connection settings 2. **Copy the encryption key** from the SDK Connection 3. **Configure the SDK** with the decryption key: ```go import ( "context" "log" gb "github.com/growthbook/growthbook-golang" ) // Create a client with encryption support client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithDecryptionKey("your_encryption_key_here"), // Add decryption key gb.WithSseDataSource(), ) if err != nil { log.Fatal("Failed to create client:", err) } defer client.Close() // Features are automatically decrypted when loaded if err := client.EnsureLoaded(context.Background()); err != nil { log.Fatal("Failed to load features:", err) } // Evaluate encrypted features normally result := client.EvalFeature(context.Background(), "sensitive-feature") fmt.Println("Feature value:", result.Value) ``` ### Loading Encrypted Features from JSON You can also load encrypted features directly from JSON: ```go // Encrypted features JSON from your API encryptedJSON := []byte(`{ "encryptedFeatures": "encrypted_payload_here...", "status": 200 }`) client, err := gb.NewClient( context.Background(), gb.WithDecryptionKey("your_encryption_key_here"), gb.WithEncryptedJsonFeature(encryptedJSON), ) if err != nil { log.Fatal("Failed to create client:", err) } // Features are decrypted and ready to use result := client.EvalFeature(context.Background(), "my-feature") ``` ### Security Best Practices ```go import ( "os" gb "github.com/growthbook/growthbook-golang" ) // Use environment variables for sensitive keys decryptionKey := os.Getenv("GROWTHBOOK_DECRYPTION_KEY") if decryptionKey == "" { log.Fatal("GROWTHBOOK_DECRYPTION_KEY environment variable not set") } client, err := gb.NewClient( context.Background(), gb.WithClientKey(os.Getenv("GROWTHBOOK_CLIENT_KEY")), gb.WithDecryptionKey(decryptionKey), gb.WithSseDataSource(), ) ``` **Security Recommendations:** - **Never hardcode** encryption keys in source code - **Use environment variables** or secret management systems (HashiCorp Vault, AWS Secrets Manager) - **Rotate keys regularly** and update across all environments - **Use different keys** for different environments (dev, staging, production) ### Handling Decryption Errors ```go client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithDecryptionKey("invalid-key"), gb.WithSseDataSource(), ) if err != nil { log.Fatal("Client initialization failed:", err) } // EnsureLoaded will fail if decryption fails ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() if err := client.EnsureLoaded(ctx); err != nil { // Log error and potentially fall back to default values log.Error("Failed to load encrypted features:", err) // Use fallback logic or default feature values } ``` :::warning Encryption Key Management If the decryption key is incorrect or missing, the SDK will fail to load features. Ensure proper error handling and monitoring to detect decryption issues in production. ::: ## Sticky Bucketing Sticky Bucketing ensures users see consistent experiment variations across sessions and devices. This is particularly useful when: - You need to slow down experiment enrollment without affecting existing users - You want to fix bugs in an experiment without including users who saw the buggy version - You need consistent experiences across different devices or sessions ### Implementation The SDK provides a built-in thread-safe in-memory implementation that you can use right away: ```go // Create an in-memory sticky bucket service service := gb.NewInMemoryStickyBucketService() // Create a client with sticky bucketing client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-XXXX"), gb.WithStickyBucketService(service), ) ``` To use sticky bucketing in an experiment, set the `BucketVersion` and optionally `MinBucketVersion` properties: ```go exp := &gb.Experiment{ Key: "my-experiment", Variations: []gb.FeatureValue{"control", "treatment"}, Meta: []gb.VariationMeta{ {Key: "0"}, // Use numeric keys to match variation IDs {Key: "1"}, }, BucketVersion: 1, // Current version of the experiment MinBucketVersion: 0, // Minimum version users must have seen HashAttribute: "userId", // Primary attribute for bucketing FallbackAttribute: "deviceId", // Used when primary is missing } result := client.RunExperiment(context.Background(), exp) ``` ### Custom Storage Implementation Implement your own persistent storage by implementing the `StickyBucketService` interface: ```go type StickyBucketService interface { GetAssignments(attributeName string, attributeValue string) (*StickyBucketAssignmentDoc, error) SaveAssignments(doc *StickyBucketAssignmentDoc) error GetAllAssignments(attributes map[string]string) (StickyBucketAssignments, error) } ``` ### Key Features 1. **Version Control**: - `BucketVersion`: Controls which version of the experiment a user is assigned to - `MinBucketVersion`: Blocks users from versions below this number 2. **Attribute-Based Bucketing**: - `HashAttribute`: Primary attribute for bucketing (usually userId) - `FallbackAttribute`: Secondary attribute when primary is missing 3. **Thread Safety**: - The in-memory implementation uses `sync.RWMutex` for concurrent access - Caching reduces database/service calls in high-traffic environments ### Experiment Result Changes The `ExperimentResult` returned by `RunExperiment` now includes a `StickyBucketUsed` boolean field that indicates if the variation was assigned from a sticky bucket: ```go result := client.RunExperiment(context.Background(), experiment) if result.StickyBucketUsed { // The user was assigned based on a previously stored assignment } ``` ## Inline Experiments Experiments can be defined and run using the `Experiment` type and the `RunExperiment` method of the client. Experiment definitions can be created directly as values of the `Experiment` type, or parsed from JSON using Go's `json.Unmarshal` function. Passing an `Experiment` value to the `RunExperiment` method will run the experiment and return an `ExperimentResult` that contains the resulting feature value. This approach allows users to run arbitrary experiments without providing feature definitions upfront. ```go experiment := &gb.Experiment{ Key: "my-experiment", Variations: []gb.FeatureValue{"red", "blue", "green"}, } result := client.RunExperiment(context.TODO(), experiment) ``` A full list of experiment fields can be found in the [documentation ](https://pkg.go.dev/github.com/growthbook/growthbook-golang#Experiment). When defining experiments, you can now use additional parameters for sticky bucketing: ```go experiment := &gb.Experiment{ Key: "my-experiment", Variations: []gb.FeatureValue{"red", "blue", "green"}, // Sticky bucket parameters BucketVersion: 1, MinBucketVersion: 0, HashAttribute: "userId", FallbackAttribute: "deviceId", } ``` ### Inline Experiment Return Value A call to `RunExperiment` returns a value of type `*ExperimentResult`: ```go result := client.RunExperiment(context.TODO(), experiment) // Whether the user is part of the experiment. fmt.Println(result.InExperiment) // true or false // The index of the assigned variation. fmt.Println(result.VariationId) // 0 or 1 // The value of the assigned variation. fmt.Println(result.Value) // "A" or "B" // The user attribute used to assign a variation. fmt.Println(result.HashAttribute) // "id" // The value of that attribute. fmt.Println(result.HashValue) // e.g., "123" // Whether a sticky bucket assignment was used fmt.Println(result.StickyBucketUsed) // true or false ``` The `InExperiment` flag is set to true only if the user was randomly assigned a variation. If the user fails any targeting rules or is forced into a specific variation, this flag will be false. ## Experiment Tracking and Feature Usage Callbacks The Go SDK provides comprehensive tracking capabilities for monitoring experiment exposures and feature evaluations. ### Experiment Tracking Callback The experiment callback is triggered when a user is included in an experiment. This is essential for analytics and experiment analysis. ```go import ( "context" "log" gb "github.com/growthbook/growthbook-golang" ) // Define your user context type UserContext struct { UserID string SessionID string } // Create experiment tracking callback experimentCallback := func(ctx context.Context, exp *gb.Experiment, result *gb.ExperimentResult, extra any) { userCtx, ok := extra.(UserContext) if !ok { log.Warn("Invalid extra data in experiment callback") return } // Track experiment view in your analytics system analytics.Track(analytics.Event{ UserID: userCtx.UserID, Event: "Experiment Viewed", Properties: map[string]interface{}{ "experiment_id": exp.Key, "variation_id": result.VariationID, "variation_value": result.Value, "in_experiment": result.InExperiment, "hash_used": result.HashUsed, "session_id": userCtx.SessionID, }, }) log.Printf("User %s entered experiment %s with variation %d", userCtx.UserID, exp.Key, result.VariationID) } // Create client with tracking client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithSseDataSource(), gb.WithExperimentCallback(experimentCallback), ) if err != nil { log.Fatal(err) } defer client.Close() ``` :::info Tracking Timing The experiment callback is **only called when the user is actually included in the experiment**. If they're excluded from the experiment due to targeting rules or sampling, this callback won't be triggered. ::: ### Feature Usage Callback The feature usage callback is called every time a feature is evaluated, regardless of whether it's part of an experiment or not. ```go // Create feature usage callback featureUsageCallback := func(ctx context.Context, featureKey string, result *gb.FeatureResult, extra any) { userCtx, ok := extra.(UserContext) if !ok { log.Warn("Invalid extra data in feature usage callback") return } // Track feature usage for monitoring and debugging log.Printf("Feature %s evaluated for user %s: %v", featureKey, userCtx.UserID, result.Value) // Send to monitoring systems (e.g., DataDog, New Relic) monitoring.RecordFeatureUsage(monitoring.FeatureUsageEvent{ FeatureKey: featureKey, Value: result.Value, Source: string(result.Source), UserID: userCtx.UserID, On: result.On, }) // Track in analytics analytics.Track(analytics.Event{ UserID: userCtx.UserID, Event: "Feature Evaluated", Properties: map[string]interface{}{ "feature_key": featureKey, "feature_value": result.Value, "source": result.Source, "is_on": result.On, }, }) } // Create client with feature usage tracking client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithSseDataSource(), gb.WithFeatureUsageCallback(featureUsageCallback), ) ``` ### Combining Both Callbacks You can use both callbacks together for comprehensive tracking: ```go type TrackingManager struct { analytics Analytics monitoring Monitoring } func (tm *TrackingManager) ExperimentCallback( ctx context.Context, exp *gb.Experiment, result *gb.ExperimentResult, extra any, ) { userCtx := extra.(UserContext) // Track in multiple systems tm.analytics.Track(analytics.Event{ UserID: userCtx.UserID, Event: "Experiment Viewed", Properties: map[string]interface{}{ "experiment_id": exp.Key, "variation_id": result.VariationID, }, }) tm.monitoring.RecordExperiment(exp.Key, result.VariationID) } func (tm *TrackingManager) FeatureUsageCallback( ctx context.Context, featureKey string, result *gb.FeatureResult, extra any, ) { userCtx := extra.(UserContext) tm.monitoring.RecordFeatureUsage(featureKey, result.Value) // Only track experiment-backed features in analytics if result.Source == gb.ExperimentResultSource { tm.analytics.Track(analytics.Event{ UserID: userCtx.UserID, Event: "Feature Used (Experiment)", Properties: map[string]interface{}{ "feature_key": featureKey, "experiment_id": result.Experiment.Key, }, }) } } // Initialize client with both callbacks func NewGrowthBookClient(tm *TrackingManager) (*gb.Client, error) { return gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithSseDataSource(), gb.WithExperimentCallback(tm.ExperimentCallback), gb.WithFeatureUsageCallback(tm.FeatureUsageCallback), ) } ``` ### Using Extra Data You can attach custom data that will be passed to each callback: ```go // Create a child client with user-specific extra data userContext := UserContext{ UserID: "user-123", SessionID: "session-abc", } childClient, err := client.WithAttributes(gb.Attributes{ "id": "user-123", }).WithExtraData(userContext) if err != nil { log.Error("Failed to create child client:", err) } // Now when features are evaluated, the callbacks will receive userContext result := childClient.EvalFeature(context.Background(), "new-checkout-flow") ``` ### Local vs Global Callbacks Callbacks can be set globally on the main client or locally on child clients: ```go // Global callback (applies to all child clients) globalClient, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithExperimentCallback(globalExperimentCallback), ) // Local callback (overrides global for this child client) childClient, err := globalClient. WithAttributes(gb.Attributes{"id": "user-123"}). WithExperimentCallback(specificExperimentCallback) // This child uses the specific callback, not the global one result := childClient.EvalFeature(context.Background(), "my-feature") ``` ### Integration Examples #### Segment.io Integration ```go import "github.com/segmentio/analytics-go" func createSegmentCallback(segmentClient analytics.Client) gb.ExperimentCallbackFunc { return func(ctx context.Context, exp *gb.Experiment, result *gb.ExperimentResult, extra any) { userID := extra.(string) segmentClient.Enqueue(analytics.Track{ UserId: userID, Event: "Experiment Viewed", Properties: analytics.NewProperties(). Set("experimentId", exp.Key). Set("variationId", result.VariationID). Set("variationValue", result.Value), }) } } client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithExperimentCallback(createSegmentCallback(segmentClient)), ) ``` #### DataDog Integration ```go import "github.com/DataDog/datadog-go/statsd" func createDataDogFeatureCallback(statsd *statsd.Client) gb.FeatureUsageCallbackFunc { return func(ctx context.Context, featureKey string, result *gb.FeatureResult, extra any) { tags := []string{ fmt.Sprintf("feature:%s", featureKey), fmt.Sprintf("source:%s", result.Source), fmt.Sprintf("on:%t", result.On), } statsd.Incr("growthbook.feature.evaluated", tags, 1) if result.On { statsd.Incr("growthbook.feature.enabled", tags, 1) } } } client, err := gb.NewClient( context.Background(), gb.WithClientKey("sdk-abc123"), gb.WithFeatureUsageCallback(createDataDogFeatureCallback(datadogClient)), ) ``` ## Logging The SDK uses the `slog` logger instance. You can set up your own logger using the `WithLogger` option when calling the `NewClient` function. It is also possible to create a child GrowthBook client with its own logger, parameterized with additional data, as shown below: ```go func main() { client, err := gb.NewClient(/* options */) if err != nil { log.Fatal(err) } // ... } func handleRequest(ctx context.Context, client *gb.Client, r request) { traceId := ctx.Value("traceId") logger := slog.Default().With("traceId", traceId) client, err := client.WithLogger(logger) if err != nil { log.Fatal(err) } client, err = client.WithAttributes(gb.Attributes{"user_id": r.user_id}) if err != nil { log.Fatal(err) } // Now, all calls to the client will use the local logger and attributes. } ``` ## Further Reading - [godoc ](https://pkg.go.dev/github.com/growthbook/growthbook-golang) --- This version addresses grammatical issues and clarifies the text while maintaining the original content and code examples. ## Supported Features --- ## GrowthBookSDKs - Feature Flags & Experimentation import { Code, CodeLogos } from '@site/src/components/Icons' import ButtonCard from '@site/src/components/ButtonCard' # SDK Overview GrowthBook SDKs are lightweight libraries that let you use feature flags and run experiments in your application. They handle feature evaluation and assignment logic. ## What SDKs Do The SDK has two main responsibilities: 1. **Fetch features from the API**: The SDK downloads your feature definitions from GrowthBook and caches them for fast access. 2. **Evaluate features**: Using attributes you provide (like user ID, country, or subscription plan), the SDK evaluates your targeting rules and returns the appropriate feature values. This evaluation happens wherever the SDK runs—in the browser, on your server, or at the edge. ## Running Experiments To measure experiment results, the SDK needs to log which variation was assigned. This is done through a tracking callback—a function you provide that sends assignment data via your event tracking system (like Segment, Mixpanel, or GA4). Without a tracking callback, the SDK will still assign variations and your features will work, but you won't be able to analyze experiment results in GrowthBook. All SDKs use the same assignment logic, so the same attributes always produce the same variation regardless of which SDK you use. Each SDK has documentation on setting up tracking callbacks, and many offer [pre-built integrations](/lib/js#plugins) with popular analytics tools. ## Choosing the Right SDK The main differences between SDK types are where they run and what problems they solve best. ### Client SDKs **Where they run:** In the browser or mobile device **Best for:** UI changes, visual experiments, and frontend feature flags **Trade-offs:** - Easy integration with frontend frameworks - Feature rules are visible to users (see [remote evaluation](/lib/js#remote-evaluation) if you need to keep rules private) - Can show a brief flicker while experiments load asynchronously (see our recommended anti-flicker techniques) **Common use cases:** Rolling out a new dashboard design, testing button colors, showing features based on user plan **SDKs:** [JavaScript](/lib/js), [React](/lib/react), [Vue](/lib/vue), [Swift](/lib/swift), [Kotlin](/lib/kotlin), [Flutter](/lib/flutter), [React Native](/lib/react-native) ### Server SDKs **Where they run:** On your backend infrastructure **Best for:** Backend logic, API responses, or when you need to keep targeting rules private **Trade-offs:** - Keeps feature logic and rules private - No visual flicker - Requires infrastructure to run (not just a script tag) - Need to pass evaluation results to the frontend if UI changes are required - May need to extract attributes from cookies, headers, or tokens **Common use cases:** Testing a new recommendation algorithm, rolling out API changes, experimenting with pricing tiers, ML model rollouts **SDKs:** [Node.js](/lib/node), [Python](/lib/python), [Ruby](/lib/ruby), [PHP](/lib/php), [Java](/lib/java), [Go](/lib/go), [C#](/lib/csharp), [Elixir](/lib/elixir) ### Edge SDKs **Where they run:** On edge platforms/CDNs (Cloudflare Workers, Fastly Compute, Lambda@Edge) **Best for:** Eliminating flicker for visual experiments or doing redirects before your application server handles the request **Trade-offs:** - Eliminates flicker for visual experiments and redirects - Can inject feature data into HTML for the frontend to use - More complex setup and integration - Resource constraints (CPU/memory limits) **Common use cases:** A/B testing landing pages without flicker, geo-based redirects, paywall experiments **SDKs:** [Cloudflare Workers](/lib/edge/cloudflare), [Fastly Compute](/lib/edge/fastly), [Lambda@Edge](/lib/edge/lambda) :::tip Not sure which to choose? **Most users start with a client SDK**. They're the easiest to implement and work great for UI changes and experiments. ::: ## Getting Started Choose your SDK below to see installation and setup instructions: ### Client } /> } /> } /> } /> } /> } /> } /> } /> } /> ### Server } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> ### Edge } /> } /> } /> --- 🛠️ **Want to build your own SDK?** Follow our [guide to integrate GrowthBook into any platform](/lib/build-your-own). --- ## Java SDK import ExternalLink from '@site/src/components/ExternalLink' import CodeTabsAddDependencyPartial from '@site/src/partials/java/_code-tabs-add-dependency.mdx' import CodeTabsGBContextBuilderPartial from '@site/src/partials/java/_code-tabs-gbcontext-builder.mdx' import CodeTabsGBContextBuilderEncryptedPartial from '@site/src/partials/java/_code-tabs-gbcontext-builder-encrypted.mdx' import CodeTabsGBContextConstructorPartial from '@site/src/partials/java/_code-tabs-gbcontext-constructor.mdx' import CodeTabsUserAttributesPartial from '@site/src/partials/java/_code-tabs-user-attributes.mdx' import CodeTabsUsingFeaturesPartial from '@site/src/partials/java/_code-tabs-using-features.mdx' import CodeTabsTrackingCallbackPartial from '@site/src/partials/java/_code-tabs-tracking-callback.mdx' import CodeTabsRunExperimentPartial from '@site/src/partials/java/_code-tabs-run-experiment.mdx' import CodeTabsExperimentResultCallbackPartial from '@site/src/partials/java/_code-tabs-subscribe-experiment-run.mdx' import CodeTabsExperimentResultSubscribeMultipleCallbackPartial from '@site/src/partials/java/_code-tabs-subscribe-multiple.mdx' import CodeTabsFeaturesRepository from '@site/src/partials/java/_code-tabs-gbfeatures-repository.mdx' import CodeTabsFeatureUsageCallback from '@site/src/partials/java/_code-tabs-feature-usage-callback.mdx' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' This supports Java applications using Java version 1.8 and higher. ## Installation [![](https://jitpack.io/v/growthbook/growthbook-sdk-java.svg)](https://jitpack.io/#growthbook/growthbook-sdk-java) ### Gradle To install in a Gradle project, add Jitpack to your repositories, and then add the dependency with the latest version to your project's dependencies. ### Maven To install in a Maven project, add Jitpack to your repositories: ```xml jitpack.io https://jitpack.io ``` Next, add the dependency with the latest version to your project's dependencies: ```xml com.github.growthbook growthbook-sdk-java 0.5.0 ``` ## Usage There are two main approaches to using the GrowthBook Java SDK: 1. **Enhanced Client (Recommended)** - For better performance with multi-context support 2. **Traditional per-request approach** - Create a new context and SDK instance per request ### Enhanced Client (Recommended) **Available starting in version 0.9.0** For improved performance and better resource management, especially in web applications, use the enhanced client pattern. This approach allows you to reuse a single client instance across multiple requests while providing different user contexts for each evaluation while calling the feature methods like `isOn()`. This `GrowthBookClient` instance is decoupled from the `GBContext`, creates a singleton featureRepository based on your refreshStrategy and uses the latest features at the time of evaluation, all managed internally. #### Basic Enhanced Client Usage ```java // build options to configure your Growthbook instance Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .build(); // Create growthbook instance using the options you need GrowthBookClient gb = new GrowthBookClient(options); // call the init method to load features gb.initialize(); gb.isOn("featureKey", UserContext.builder() .attributesJson("{\"id\" : \"123\"}").build() ); ``` #### Using the Enhanced Client with Different User Contexts ```java // build options to configure your Growthbook instance Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .build(); // Create growthbook instance using the options you need GrowthBookClient gb = new GrowthBookClient(options); // call the init method to load features gb.initialize(); // Use the same client instance with different user contexts // User 1 context UserContext user1Context = UserContext.builder() .attributesJson("{\"id\":\"user_123\",\"country\":\"US\",\"premium\":true}") .build(); boolean showNewFeature = gb.isOn("new-homepage", user1Context); String buttonColor = gb.getFeatureValue("button-color", "blue", user1Context); // User 2 context - same client, different context UserContext user2Context = UserContext.builder() .attributesJson("{\"id\":\"user_456\",\"country\":\"CA\",\"premium\":false}") .build(); boolean showFeatureForUser2 = gb.isOn("new-homepage", user2Context); String colorForUser2 = gb.getFeatureValue("button-color", "blue", user2Context); ``` #### Thread Safety Note While the enhanced client is designed for concurrent use, full concurrency support is still being refined. For high-concurrency applications, consider: - Using connection pooling for database-backed sticky bucket services - Implementing proper synchronization for custom tracking callbacks - Testing thoroughly under expected load conditions ### Traditional Usage **For new projects, we recommend using the [Enhanced Client](#enhanced-client-recommended) instead for better performance.** There are 2 steps to initializing the traditional GrowthBook SDK: 1. Create a GrowthBook context `GBContext` with the features JSON and the user attributes 2. Create the `GrowthBook` SDK class with the context ### GrowthBook context The GrowthBook context `GBContext` can be created either by implementing the builder class, available at `GBContext.builder()`, or by using the `GBContext` constructor. | Field name | Type | Description | | ---------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `attributesJson` | `String` | The user attributes JSON. See [Attributes](#attributes). | | `featuresJson` | `String` | The features JSON served by the GrowthBook API (or equivalent). See [Features](#features). | | `enabled` | `Boolean` | Whether to enable the functionality of the SDK (default: `true`) | | `isQaMode` | `Boolean` | Whether the SDK is in QA mode. Not for production use. If true, random assignment is disabled and only explicitly forced variations are used (default: `false`) | | `url` | `String` | The URL of the current page. Useful when evaluating features and experiments based on the page URL. | | `forcedVariationsMap` | `Map` | Force specific experiments to always assign a specific variation (used for QA) | | `trackingCallback` | `TrackingCallback` | A callback that will be invoked with every experiment evaluation where the user **is** included in the experiment. See [TrackingCallback](#tracking-callback). To subscribe to all evaluated events regardless of whether the user is in the experiment, see [Subscribing to experiment runs](#subscribing-to-experiment-runs-with-the-experimentruncallback). | | `featureUsageCallback` | `FeatureUsageCallback` | A callback that will be invoked every time a feature is viewed. See [FeatureUsageCallback](#feature-usage-callback) | #### Using the GBContext builder The builder is the easiest to use way to construct a `GBContext`, allowing you to provide as many or few arguments as you'd like. All fields mentioned above are available via the builder. :::info HttpClient recommendation The above example uses `java.net.http.HttpClient` which, depending on your web framework, may not be the best option, in which case it is recommended to use a networking library more suitable for your implementation. ::: #### Using the GBContext constructor You can also use `GBContext` constructor if you prefer, which will require you to pass all arguments explicitly. For complete examples, see the [Examples](#code-examples) section below. #### Features The features JSON is equivalent to the `features` property that is returned from the SDK Connection endpoint. - You can read more [about features here](/app/features) - You can see an [example features JSON here](/app/api#sdk-connection-endpoints) #### Attributes Attributes are a JSON string. You can specify attributes about the current user and request. Here's an example: If you need to set or update attributes asynchronously, you can do so with `Context#attributesJson` or `GrowthBook#setAttributes`. This will completely overwrite the attributes object with whatever you pass in. Also, be aware that changing attributes may change the assigned feature values. This can be disorienting to users if not handled carefully. #### Secure Attributes When _secure attribute hashing_ is enabled, all targeting conditions in the SDK payload referencing attributes with datatype `secureString` or `secureString[]` will be anonymized via SHA-256 hashing. This allows you to safely target users based on sensitive attributes. You must enable this feature in your SDK Connection for it to take effect. If your SDK Connection has secure attribute hashing enabled, you will need to manually hash any `secureString` or `secureString[]` attributes that you pass into the GrowthBook SDK. To hash an attribute, use a cryptographic library with SHA-256 support, and compute the SHA-256 hashed value of your attribute _plus_ your organization's secure attribute salt. ```mdx-code-block ``` ```java import java.nio.charset.StandardCharsets; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; import java.util.Base64; // Your secure attribute salt (set in Organization Settings) String salt = "f09jq3fij"; // Create and initialize the enhanced client once Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .build(); GrowthBookClient gb = new GrowthBookClient(options); gb.initialize(); // Helper method to hash secure attributes private String hashSecureAttribute(String value, String salt) { try { MessageDigest digest = MessageDigest.getInstance("SHA-256"); byte[] hash = digest.digest((salt + value).getBytes(StandardCharsets.UTF_8)); return Base64.getEncoder().encodeToString(hash); } catch (NoSuchAlgorithmException e) { throw new RuntimeException("SHA-256 not available", e); } } // Hash secure attributes for each user evaluation String userEmail = user.getEmail(); String hashedEmail = hashSecureAttribute(userEmail, salt); List userTags = user.getTags(); List hashedTags = userTags.stream() .map(tag -> hashSecureAttribute(tag, salt)) .collect(Collectors.toList()); // Create user context with hashed attributes UserContext userContext = UserContext.builder() .attributesJson(String.format( "{\"id\":\"%s\",\"loggedIn\":true,\"email\":\"%s\",\"tags\":%s}", user.getId(), hashedEmail, new Gson().toJson(hashedTags) )) .build(); // Use the enhanced client with hashed attributes boolean hasFeature = gb.isOn("premium-feature", userContext); String theme = gb.getFeatureValue("theme", "light", userContext); ``` ```mdx-code-block ``` ```java import java.nio.charset.StandardCharsets; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; import java.util.Base64; // Your secure attribute salt (set in Organization Settings) String salt = "f09jq3fij"; // Hashing a secureString attribute String userEmail = user.getEmail(); String hashedEmail = ""; try { MessageDigest digest = MessageDigest.getInstance("SHA-256"); byte[] hash = digest.digest((salt + userEmail).getBytes(StandardCharsets.UTF_8)); hashedEmail = Base64.getEncoder().encodeToString(hash); } catch (NoSuchAlgorithmException e) { // Handle exception } // Hashing a secureString[] attribute List userTags = user.getTags(); List hashedTags = new ArrayList(); try { MessageDigest digest = MessageDigest.getInstance("SHA-256"); for (String tag : userTags) { byte[] hash = digest.digest((salt + tag).getBytes(StandardCharsets.UTF_8)); hashedTags.add(Base64.getEncoder().encodeToString(hash)); } } catch (NoSuchAlgorithmException e) { // Handle exception } // Create attributes JSON with hashed values String attributesJson = String.format( "{\"id\":\"%s\",\"loggedIn\":true,\"email\":\"%s\",\"tags\":%s}", user.getId(), hashedEmail, Arrays.toString(hashedTags.toArray()) ); // Use the hashed attributes when creating your GrowthBook instance GBContext context = GBContext .builder() .featuresJson(featuresJson) .attributesJson(attributesJson) .build(); GrowthBook growthBook = new GrowthBook(context); ``` ```mdx-code-block ``` ### Using Features Every feature has a "value" which is assigned to a user. This value can be any JSON data type. If a feature doesn't exist, the value will be `null`. There are 4 main methods for evaluating features. | Method | Return type | Description | | ----------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------- | | **`isOn(String)`** | `Boolean` | Returns true if the value is a truthy value | | **`isOff(String)`** | `Boolean` | Returns true if the value is a falsy value | | **`getFeatureValue(String)`** | generic `T` (nullable) | Returns the value cast to the generic type. Type is inferred based on the `defaultValue` argument provided. | | **`evalFeature(String)`** | `FeatureResult` | Returns a feature result with a value of generic type `T`. The value type needs to be specified in the generic parameter. | ```mdx-code-block ``` ```java // Create enhanced client Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .build(); GrowthBookClient gb = new GrowthBookClient(options); gb.initialize(); // Create user context for evaluations UserContext userContext = UserContext.builder() .attributesJson("{\"id\":\"user_123\",\"premium\":true}") .build(); // Boolean feature checks if (gb.isOn("dark_mode", userContext)) { // value is truthy } if (gb.isOff("dark_mode", userContext)) { // value is falsy } // Get feature values with defaults Float donutPrice = gb.getFeatureValue("donut_price", 5.0f, userContext); String theme = gb.getFeatureValue("theme", "light", userContext); Integer maxItems = gb.getFeatureValue("max_items", 10, userContext); // Get detailed feature result FeatureResult priceResult = gb.evalFeature("donut_price", userContext); Float price = priceResult.getValue(); String source = priceResult.getSource().toString(); ``` ```mdx-code-block ``` ```java // Create traditional client GBContext context = GBContext .builder() .featuresJson(featuresJson) .attributesJson("{\"id\":\"user_123\",\"premium\":true}") .build(); GrowthBook growthBook = new GrowthBook(context); // Boolean feature checks if (growthBook.isOn("dark_mode")) { // value is truthy } if (growthBook.isOff("dark_mode")) { // value is falsy } // Get feature values with defaults Float donutPrice = growthBook.getFeatureValue("donut_price", 5.0f); String theme = growthBook.getFeatureValue("theme", "light"); Integer maxItems = growthBook.getFeatureValue("max_items", 10); // Get detailed feature result FeatureResult priceResult = growthBook.evalFeature("donut_price"); Float price = priceResult.getValue(); String source = priceResult.getSource().toString(); ``` ```mdx-code-block ``` #### isOn() / isOff() These methods return a boolean for truthy and falsy values. Only the following values are considered to be "falsy": - `null` - `false` - `""` - `0` Everything else is considered "truthy", including empty arrays and objects. If the value is "truthy", then `isOn()` will return true and `isOff()` will return false. If the value is "falsy", then the opposite values will be returned. #### getFeatureValue(featureKey, defaultValue) This method has a variety of overloads to help with casting values to primitive and complex types. In short, the type of the `defaultValue` argument will determine the return type of the function. | Return type | Method | Additional Info | | ----------------------- | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Boolean` | **`getFeatureValue(String featureKey, Boolean defaultValue)`** | | | `Double` | **`getFeatureValue(String featureKey, Double defaultValue)`** | | | `Float` | **`getFeatureValue(String featureKey, Float defaultValue)`** | | | `Integer` | **`getFeatureValue(String featureKey, Integer defaultValue)`** | | | `String` | **`getFeatureValue(String featureKey, String defaultValue)`** | | | ` ValueType` | **`getFeatureValue(String featureKey, ValueType defaultValue, Class gsonDeserializableClass)`** | Internally, the SDK uses Gson. You can pass any class that does not require a custom deserializer. | | `Object` | **`getFeatureValue(String featureKey, Object defaultValue)`** | Use this method if you need to cast a complex object that uses a custom deserializer, or if you use a different JSON serialization library than Gson, and cast the type yourself. | [See the Java Docs](https://growthbook.github.io/growthbook-sdk-java/growthbook/sdk/java/GrowthBook.html) for more information. [See the unit tests ](https://github.com/growthbook/growthbook-sdk-java/blob/main/lib/src/test/java/growthbook/sdk/java/GrowthBookTest.java#L220) for example implementations including type casting for all above-mentioned methods. #### evalFeature(String) The `evalFeature` method returns a `FeatureResult` object with more info about why the feature was assigned to the user. The `T` type corresponds to the value type of the feature. In the above example, `T` is `Float`. `FeatureResult` It has the following getters. | Method | Return type | Description | | --------------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------- | | **`getValue()`** | generic `T` (nullable) | The evaluated value of the feature | | **`getSource()`** | `enum FeatureResultSource` (nullable) | The reason/source for the evaluated feature value. | | **`getRuleId()`** | `String` (nullable) | ID of the rule that was used to assign the value to the user. | | **`getExperiment()`** | `Experiment` (nullable) | The experiment details, available only if the feature evaluates due to an experiment. | | **`getExperimentResult()`** | `ExperimentResult` (nullable) | The experiment result details, available only if the feature evaluates due to an experiment. | As expected in Kotlin, you can access these getters using property accessors. ### Inline Experiments Instead of declaring all features up-front in the context and referencing them by IDs in your code, you can also just run an experiment directly. This is done with the `growthbook.run(Experiment)` method. ```mdx-code-block ``` ```java // Create enhanced client Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .build(); GrowthBookClient gb = new GrowthBookClient(options); gb.initialize(); // Create user context UserContext userContext = UserContext.builder() .attributesJson("{\"id\":\"user_123\",\"employee\":true}") .build(); // Create and run inline experiment Experiment donutPriceExperiment = Experiment .builder() .key("donut-price-test") .variations(Arrays.asList(2.50f, 3.00f, 3.50f)) .weights(Arrays.asList(0.33f, 0.33f, 0.34f)) .conditionJson("{\"employee\": true}") .build(); ExperimentResult result = gb.run(donutPriceExperiment, userContext); Float donutPrice = result.getValue(); // Check if user was included in experiment if (result.getInExperiment()) { System.out.println("User is in experiment, variation: " + result.getVariationId()); System.out.println("Donut price: $" + donutPrice); } else { System.out.println("User not in experiment"); } // String experiment example Experiment buttonColorExperiment = Experiment .builder() .key("button-color") .variations(Arrays.asList("blue", "red", "green")) .build(); ExperimentResult colorResult = gb.run(buttonColorExperiment, userContext); String buttonColor = colorResult.getValue(); ``` ```mdx-code-block ``` ```java // Create traditional client GBContext context = GBContext .builder() .featuresJson(featuresJson) .attributesJson("{\"id\":\"user_123\",\"employee\":true}") .build(); GrowthBook growthBook = new GrowthBook(context); // Create and run inline experiment Experiment donutPriceExperiment = Experiment .builder() .key("donut-price-test") .variations(Arrays.asList(2.50f, 3.00f, 3.50f)) .weights(Arrays.asList(0.33f, 0.33f, 0.34f)) .conditionJson("{\"employee\": true}") .build(); ExperimentResult result = growthBook.run(donutPriceExperiment); Float donutPrice = result.getValue(); // Check if user was included in experiment if (result.getInExperiment()) { System.out.println("User is in experiment, variation: " + result.getVariationId()); System.out.println("Donut price: $" + donutPrice); } else { System.out.println("User not in experiment"); } // String experiment example Experiment buttonColorExperiment = Experiment .builder() .key("button-color") .variations(Arrays.asList("blue", "red", "green")) .build(); ExperimentResult colorResult = growthBook.run(buttonColorExperiment); String buttonColor = colorResult.getValue(); ``` ```mdx-code-block ``` #### Inline experiment return value ExperimentResult An `ExperimentResult` is returned where `T` is the generic value type for the experiment. There's also a number of methods available. | Method | Return type | Description | | ------------------------ | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **`getValue()`** | generic `T` (nullable) | The evaluated value of the feature | | **`getVariationId()`** | `Integer` (nullable) | Index of the variation used, if applicable | | **`getInExperiment()`** | `Boolean` | If the user was in the experiment. This will be false if the user was excluded from being part of the experiment for any reason (e.g. failed targeting conditions). | | **`getHashAttribute()`** | `String` | User attribute used for hashing, defaulting to `id` if not set. | | **`getHashValue()`** | `String` (nullable) | The hash value used for evaluating the experiment, if applicable. | | **`getFeatureId()`** | `String` | The feature key/ID | | **`getHashUsed()`** | `Boolean` | If a hash was used to evaluate the experiment. This flag will only be true if the user was randomly assigned a variation. If the user was forced into a specific variation instead, this flag will be false. | As expected in Kotlin, you can access these getters using property accessors. ### Tracking feature usage and experiment impressions This section covers how to track and monitor feature usage and experiment impressions in your application. There are several types of callbacks and events you can subscribe to: 1. **Tracking Callback** - Called when a user is included in an experiment 2. **Feature Usage Callback** - Called every time a feature is evaluated 3. **Experiment Run Callback** - Called for every experiment evaluation (regardless of inclusion) #### Tracking Callback Any time an experiment is run to determine the value of a feature, we may call this callback so you can record the assigned value in your event tracking or analytics system of choice. **The tracking callback is only called when the user is in the experiment**. If they are not in the experiment, this will not be called. ```mdx-code-block ``` ```java // Create tracking callback TrackingCallback trackingCallback = new TrackingCallback() { @Override public void onTrack( Experiment experiment, ExperimentResult experimentResult ) { // Send to your analytics system analytics.track("experiment_viewed", Map.of( "experiment_id", experiment.getKey(), "variation_id", experimentResult.getVariationId(), "user_id", experimentResult.getHashAttribute() )); } }; // Create enhanced client with tracking Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .trackingCallback(trackingCallback) .build(); GrowthBookClient gb = new GrowthBookClient(options); gb.initialize(); // Use features - tracking callback will be called automatically when user is in experiment UserContext userContext = UserContext.builder() .attributesJson("{\"id\":\"user_123\",\"employee\":true}") .build(); boolean newFeature = gb.isOn("new-checkout-flow", userContext); ``` ```mdx-code-block ``` ```java // Create tracking callback TrackingCallback trackingCallback = new TrackingCallback() { @Override public void onTrack( Experiment experiment, ExperimentResult experimentResult ) { // Send to your analytics system System.out.println("Experiment tracked: " + experiment.getKey()); System.out.println("Variation: " + experimentResult.getVariationId()); } }; // Create context with tracking callback GBContext context = GBContext .builder() .featuresJson(featuresJson) .attributesJson(userAttributesJson) .trackingCallback(trackingCallback) .build(); GrowthBook growthBook = new GrowthBook(context); // Use features - tracking callback will be called when user is in experiment boolean newFeature = growthBook.isOn("new-checkout-flow"); ``` ```mdx-code-block ``` #### Feature Usage Callback Any time a feature is evaluated (regardless of experiment participation), this callback is called with the feature key and result. ```mdx-code-block ``` ```java // Create feature usage callback FeatureUsageCallback featureUsageCallback = new FeatureUsageCallback() { @Override public void onFeatureUsage( String featureKey, FeatureResult featureResult ) { // Log feature usage logger.info("Feature '{}' evaluated with value: {}", featureKey, featureResult.getValue()); // Send to analytics analytics.track("feature_used", Map.of( "feature_key", featureKey, "feature_value", featureResult.getValue(), "source", featureResult.getSource().toString() )); } }; // Create enhanced client with feature usage tracking Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .featureUsageCallback(featureUsageCallback) .build(); GrowthBookClient gb = new GrowthBookClient(options); gb.initialize(); // Feature usage callback will be called for every feature evaluation UserContext userContext = UserContext.builder() .attributesJson("{\"id\":\"user_123\"}") .build(); String theme = gb.getFeatureValue("theme", "light", userContext); boolean darkMode = gb.isOn("dark_mode", userContext); ``` ```mdx-code-block ``` ```java // Create feature usage callback FeatureUsageCallback featureUsageCallback = new FeatureUsageCallback() { @Override public void onFeatureUsage( String featureKey, FeatureResult featureResult ) { // Log feature usage System.out.println("Feature used: " + featureKey + " = " + featureResult.getValue()); } }; // Create context with feature usage callback GBContext context = GBContext .builder() .featuresJson(featuresJson) .attributesJson(userAttributesJson) .featureUsageCallback(featureUsageCallback) .build(); GrowthBook growthBook = new GrowthBook(context); // Feature usage callback will be called for every feature evaluation String theme = growthBook.getFeatureValue("theme", "light"); boolean darkMode = growthBook.isOn("dark_mode"); ``` ```mdx-code-block ``` #### Experiment Run Callback You can subscribe to experiment run evaluations using the `ExperimentRunCallback`. **This callback is called for every experiment evaluation, regardless of whether the user is included in the experiment**. ```mdx-code-block ``` ```java // Create experiment run callback ExperimentRunCallback experimentRunCallback = new ExperimentRunCallback() { @Override public void onExperimentRun( Experiment experiment, ExperimentResult experimentResult ) { // Track all experiment evaluations (included and excluded users) analytics.track("experiment_evaluated", Map.of( "experiment_id", experiment.getKey(), "user_in_experiment", experimentResult.getInExperiment(), "variation_id", experimentResult.getVariationId(), "hash_used", experimentResult.getHashUsed() )); } }; // Enhanced client doesn't directly support ExperimentRunCallback // Use inline experiments with the callback Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .build(); GrowthBookClient gb = new GrowthBookClient(options); gb.initialize(); // Subscribe to experiment runs (you can subscribe multiple callbacks) gb.subscribe(experimentRunCallback); // Run inline experiments - callback will be triggered UserContext userContext = UserContext.builder() .attributesJson("{\"id\":\"user_123\",\"employee\":true}") .build(); Experiment buttonColorExperiment = Experiment .builder() .key("button-color") .variations(Arrays.asList("blue", "red", "green")) .build(); ExperimentResult result = gb.run(buttonColorExperiment, userContext); ``` ```mdx-code-block ``` ```java // Create experiment run callback ExperimentRunCallback experimentRunCallback = new ExperimentRunCallback() { @Override public void onExperimentRun( Experiment experiment, ExperimentResult experimentResult ) { // Track all experiment evaluations System.out.println("Experiment evaluated: " + experiment.getKey()); System.out.println("User in experiment: " + experimentResult.getInExperiment()); } }; // Create GrowthBook instance GBContext context = GBContext .builder() .featuresJson(featuresJson) .attributesJson(userAttributesJson) .build(); GrowthBook growthBook = new GrowthBook(context); // Subscribe to experiment runs (you can subscribe multiple callbacks) growthBook.subscribe(experimentRunCallback); // Run inline experiments - callback will be triggered Experiment buttonColorExperiment = Experiment .builder() .key("button-color") .variations(Arrays.asList("blue", "red", "green")) .build(); ExperimentResult result = growthBook.run(buttonColorExperiment); ``` ```mdx-code-block ``` #### Multiple Callback Subscriptions You can subscribe to multiple experiment run callbacks: ```mdx-code-block ``` ```java // Multiple callbacks for different purposes ExperimentRunCallback analyticsCallback = (experiment, result) -> { // Send to analytics analytics.track("experiment_run", Map.of( "experiment", experiment.getKey(), "included", result.getInExperiment() )); }; ExperimentRunCallback loggingCallback = (experiment, result) -> { // Log for debugging logger.debug("Experiment {} evaluated for user {}", experiment.getKey(), result.getHashAttribute()); }; // Subscribe multiple callbacks gb.subscribe(analyticsCallback); gb.subscribe(loggingCallback); ``` ```mdx-code-block ``` ```java // Multiple callbacks for different purposes ExperimentRunCallback analyticsCallback = (experiment, result) -> { // Send to analytics System.out.println("Analytics: " + experiment.getKey()); }; ExperimentRunCallback loggingCallback = (experiment, result) -> { // Log for debugging System.out.println("Debug: " + experiment.getKey()); }; // Subscribe multiple callbacks growthBook.subscribe(analyticsCallback); growthBook.subscribe(loggingCallback); ``` ```mdx-code-block ``` ### Working with Encrypted features As of version 0.3.0, the Java SDK supports decrypting encrypted features. You can learn more about [SDK Connection Endpoint Encryption](/app/api#encryption). The main difference is you create a `GBContext` by passing an encryption key (`.encryptionKey()` when using the builder) and using the encrypted payload as the features JSON (`.featuresJson()` for the builder). ### Fetching, Caching, and Refreshing features with GBFeaturesRepository As of version 0.4.0, the Java SDK provides an optional `GBFeaturesRepository` class which will manage networking for you in the following ways: - Fetching features from the SDK endpoint when `initialize()` is called - Decrypting encrypted features when provided with the client key, e.g. `.builder().encryptionKey(clientKey)` - Caching features (in-memory) - Refreshing features If you wish to manage fetching, refreshing, and caching features on your own, you can choose to not implement this class. :::info Recommendation This class should be implemented as a singleton class as it includes caching and refreshing functionality. ::: If you have more than one SDK endpoint you'd like to implement, you can extend the `GBFeaturesRepository` class with your own class to make it easier to work with dependency injection frameworks. Each of these instances should be singletons. #### Fetching the features You will need to create a singleton instance of the `GBFeaturesRepository` class either by implementing its `.builder()` or by using its constructor. Then, you would call `myGbFeaturesRepositoryInstance.initialize()` in order to make the initial (blocking) request to fetch the features. Then, you would call `myGbFeaturesRepositoryInstance.getFeaturesJson()` and provided that to the `GBContext` initialization. For more references, see the [Examples](#code-examples) below. #### Caching and refreshing behavior As of version 0.9.0, there are 2 refresh strategies available. ##### Stale While Revalidate This is the default strategy but can be explicitly stated by passing `FeatureRefreshStrategy.STALE_WHILE_REVALIDATE` as the refresh strategy option to the `GBFeaturesRepository` builder or constructor. The `GBFeaturesRepository` will automatically refresh the features when the features become stale. Features are considered stale every 60 seconds. This amount is configurable with the `ttlSeconds` option. When you fetch features and they are considered stale, the stale features are returned from the `getFeaturesJson()` method and a network call to refresh the features is enqueued asynchronously. When that request succeeds, the features are updated with the newest features, and the next call to `getFeaturesJson()` will return the refreshed features. ##### Server-Sent Events This is a new strategy that can be enabled by passing `FeatureRefreshStrategy.SERVER_SENT_EVENTS` as the refresh strategy option to the `GBFeaturesRepository` builder or constructor. If you're using [GrowthBook Cloud ](https://app.growthbook.io), this is ready for you to use. If you are self-hosting, you will need to set up the [GrowthBook Proxy](/self-host/proxy) to enable it. ## Overriding Feature Values The Java SDK allows you to override feature values and experiments using the URL. ### Force Experiment Variations You can force an experiment variation by passing the experiment key and variation index as query parameters in the URL you set on the `GBContext`. For example, if you add `?my-experiment-id=2` to the URL, users will be forced into the variation at index 2 in the variations list when evaluating the experiment with key `my-experiment-id`. ### Force Feature Values via the URL You can force a value for a feature by passing the key, prefixed by `gb~`, and the URI-encoded value in the URL's query parameters. You must also set `allowUrlOverrides` to true when building your `GBContext` in order to enable this feature as it is not enabled by default. ```java GBContext context = GBContext .builder() .url("http://localhost:8080/url-feature-force?gb~dark_mode=true&gb~donut_price=3.33&gb~banner_text=Hello%2C%20everyone!%20I%20hope%20you%20are%20all%20doing%20well!") .allowUrlOverrides(true) .build(); ``` The above code sample sets the following: - `dark_mode`: `true` - `banner_text`: `"Hello, everyone! I hope you are all doing well!"` - `donut_price`: `3.33` #### Supported types | Type | Value | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `String` | A URI-encoded string value | | `Float` | A float value, e.g. `3.33` | | `Double` | A double value, e.g. `3.33` | | `Integer` | An integer value, e.g. `1337` | | `Boolean` | A boolean value, e.g. `true` or `false`. You can also represent boolean values with on/off state as `on` or `off` or binary values `1` or `0`. | | JSON as `Gson`-deserializable | A class that can be deserialized using Gson and that does not have any dependencies on custom type adapters. | | JSON as `String` | A URI-encoded string value that should be a valid JSON string that you can deserialize with your own JSON deserialization implementation. | The value passed in the URL is cast at runtime based on the generic type argument passed in when evaluating the feature. This means that when you call `getFeatureValue()`, what you pass into the URL must successfully cast as `ValueType` otherwise the value in the URL will be ignored. All keys must be prefixed with `gb~`. ## Using with Proguard and R8 Many Android projects use code-shrinking and obfuscation tools like Proguard and R8 in production. If you are experiencing unexpected feature evaluation results with your release Android builds that do not occur in your debug builds, it's most likely related to this. You will need to add the following to your `proguard-rules.pro` file to ensure that all of the GrowthBook SDK classes are kept so that your features are evaluated properly in projects that use Proguard and R8: ``` # Growthbook Java SDK classes -keep class growthbook.sdk.java.** { *; } ``` ## Code Examples ### Web Framework Integration with Spring RestController ```java @RestController public class FeatureController { // Singleton enhanced client instance private final GrowthBookClient growthBookClient; @Autowired public FeatureController() { // build options to configure your Growthbook instance Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .build(); // Create growthbook instance using the options you need this.growthBookClient = new GrowthBookClient(options); // Initialize features once this.growthBookClient.initialize(); } @GetMapping("/api/features") public ResponseEntity> getFeatures(HttpServletRequest request) { // Create user context per request UserContext userContext = createUserContext(request); // Evaluate features using shared client Map features = new HashMap(); features.put("darkMode", growthBookClient.isOn("dark-mode", userContext)); features.put("maxItems", growthBookClient.getFeatureValue("max-items", 10, userContext)); features.put("newLayout", growthBookClient.isOn("new-layout", userContext)); return ResponseEntity.ok(features); } @GetMapping("/api/user/{userId}/features") public ResponseEntity> getUserFeatures(@PathVariable String userId) { // Create user-specific context UserContext userContext = UserContext.builder() .attributesJson(String.format("{\"id\":\"%s\",\"loggedIn\":true}", userId)) .build(); // Evaluate features for specific user Map features = new HashMap(); features.put("premiumFeature", growthBookClient.isOn("premium-feature", userContext)); features.put("theme", growthBookClient.getFeatureValue("theme", "light", userContext)); return ResponseEntity.ok(features); } private UserContext createUserContext(HttpServletRequest request) { // Extract user attributes from request String userId = getUserIdFromSession(request); String userAgent = request.getHeader("User-Agent"); String country = getCountryFromRequest(request); // Build attributes JSON String attributesJson = String.format( "{\"id\":\"%s\",\"userAgent\":\"%s\",\"country\":\"%s\",\"loggedIn\":%s}", userId, userAgent, country, userId != null ); return UserContext.builder() .attributesJson(attributesJson) .build(); } private String getUserIdFromSession(HttpServletRequest request) { // Your implementation to extract user ID from session/JWT/etc return request.getSession().getAttribute("userId") != null ? request.getSession().getAttribute("userId").toString() : null; } private String getCountryFromRequest(HttpServletRequest request) { // Your implementation to determine user's country return request.getHeader("CF-IPCountry"); // Example using Cloudflare header } } ``` ### Additional Examples - [JVM with Spring Web example ](https://github.com/growthbook/examples/tree/main/jvm-spring-web) - [JVM example in Kotlin with Ktor ](https://github.com/growthbook/examples/tree/main/jvm-kotlin-ktor-example) - [Android Java example ](https://github.com/growthbook/examples/tree/main/android-example) ## Further Reading - [JavaDoc class documentation ](https://growthbook.github.io/growthbook-sdk-java/) ## Supported Features --- ## Javascript SDK import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Javascript The documentation is meant for client-side environments. Check out the [Node.js docs](/lib/node) for server-side environments. ## Installation Install with a package manager ```bash npm2yarn npm install --save @growthbook/growthbook ``` ## Quick Usage ### Step 1: Configure your app ```js import { GrowthBook } from "@growthbook/growthbook"; // Create a GrowthBook instance const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", // Targeting attributes attributes: { id: "123", country: "US" }, // Only required for A/B testing // Called every time a user is put into an experiment trackingCallback: (experiment, result, user) => { console.log("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, attributes: user?.attributes, }); }, }); // Download features and experiments from the CDN // Also, start running any Visual Editor or URL Redirect experiments await gb.init(); ``` ### Step 2: Start Feature Flagging! There are 2 main methods for evaluating features: `isOn` and `getFeatureValue`: ```js // Simple boolean (on/off) feature flag if (gb.isOn("my-feature")) { console.log("Feature enabled!"); } // Get the value of a string/JSON/number feature with a fallback const color = gb.getFeatureValue("button-color", "blue"); ``` ## Plugins GrowthBook comes with a number of built-in plugins that add additional functionality. You can enable these by passing them into the GrowthBook constructor. Plugins require GrowthBook version 1.4.0 or higher, released in Feb 2025. ### Third Party Tracking The third-party tracking plugin automatically sends an "Experiment Viewed" event to several popular analytics tracking tools (Segment, Google Analytics, and Google Tag Manager). This plugin requires a browser environment and does not work from Node.js. This plugin does not install any 3rd party libraries, it just sends events to ones that are already installed on your site. Below are all of the supported trackers: - `segment` - Requires `window.analytics` to be defined. - `gtag` - Requires `window.gtag` to be defined. - `gtm` - Requires `window.dataLayer` to be defined. ```js import { thirdPartyTrackingPlugin } from "@growthbook/growthbook/plugins"; // Optional settings for the plugin const pluginOptions = { // By default, it will attempt to send to all 3 of these trackers: ["segment", "gtag", "gtm"], // Additional custom tracking callback additionalCallback: (experiment, result) => { console.log("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, }) } } const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", plugins: [ thirdPartyTrackingPlugin(pluginOptions), ], }); ``` ### Auto-Attributes The auto-attributes plugin automatically detects and sets common attributes in GrowthBook for the current user. These attributes are available locally for targeting and are not sent to GrowthBook's servers. This plugin requires a browser environment and does not work from Node.js. ```js import { autoAttributesPlugin } from "@growthbook/growthbook/plugins"; // Optional settings for the plugin (documented below) const pluginOptions: {} const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", plugins: [ autoAttributesPlugin(pluginOptions), ], }); ``` #### Attributes Here is a list of all attributes that are set by this plugin: - `id` - A random unique identifier for the user, persisted in a cookie - `url` - The current URL of the page - `path` - The current URL path of the page - `host` - The current URL host of the page - `query` - The current URL query string of the page - `pageTitle` - The current page title - `deviceType` - The device type of the user. Either "mobile" or "desktop" - `browser` - The browser name of the user (edge, chrome, firefox, safari, or unknown) - `utmSource`, `utmMedium`, `utmCampaign`, `utmTerm`, `utmContent` - UTM parameters from the URL (persisted in sessionStorage for subsequent page views) In addition, any variables set through Google Tag Manager (`window.dataLayer`) will be added as attributes as well. You can still specify your own custom attributes in the GrowthBook constructor, and they will be merged with the auto-attributes. #### Options By default the plugin generates a random UUID for the user and stores it in a cookie. You can customize this behavior with the following options. - `uuidKey` - The attribute key to use for the uuid (default is `id`) - `uuid` - Pass in your own user id instead of having the plugin generate a random one. - `uuidCookieName` - The name of the cookie to use for the generated uuid (default is `gb_uuid`) - `uuidCookieDomain` - Scope the uuid cookie to a parent domain (e.g. `".example.com"`) so the same anonymous id is shared across subdomains. Recommended if you run experiments that redirect users between subdomains, otherwise the visitor gets a new id (and can be re-bucketed) on the destination subdomain. - `uuidAutoPersist` - If `false`, the generated id will NOT be stored in a cookie automatically. Default depends on the `uuid` option. If `uuid` is set, the default is `false`. Otherwise, the default is `true`. If you have a cookie consent banner, you can set `uuidAutoPersist` to `false` and then later manually trigger the cookie to be saved by calling: ```js document.dispatchEvent(new CustomEvent("growthbookpersist")); ``` ## Loading Features and Experiments In order for the GrowthBook SDK to work, it needs to have feature and experiment definitions from the GrowthBook API. There are a few ways to get this data into the SDK. ### Built-in Fetching and Caching If you pass an `apiHost` and `clientKey` into the GrowthBook constructor, it will handle the network requests, caching, retry logic, etc. for you automatically. If your feature payload is encrypted, you can also pass in a `decryptionKey`. ```ts const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", // Only required if you have feature encryption enabled in GrowthBook decryptionKey: "key_abc123", }); // Wait for features to be downloaded with a timeout (in ms) await gb.init({ timeout: 2000, }); ``` Until features are loaded, all features will evaluate to `null`. If you're ok with a potential flicker in your application (features going from `null` to their real value), you can call `init` without awaiting the result. If you want to refresh the features at any time (e.g. when a navigation event occurs), you can call `gb.refreshFeatures()`. #### Error Handling In the case of network issues, the `init` call will not throw an error. Instead, it will stay in the default state where every feature evaluates to `null`. You can still get access to the error if needed: ```ts const res = await gb.init({ timeout: 1000 }); console.log(res); ``` The return value has 3 properties: - **success** - `true` if the GrowthBook instance was populated with features/experiments. Otherwise `false` - **source** - Where this result came from. One of the following values: `network`, `cache`, `init`, `error`, or `timeout` - **error** - If success is `false`, this will contain an `Error` object with more details about the error ### Custom Integration If you prefer to handle the network and caching logic yourself, you can pass in a full JSON "payload" directly into the SDK. For example, you might store features in Postgres and send it down to your front-end as part of your app's initial bootstrap API call. ```ts await gb.init({ payload: { features: { "feature-1": {...}, "feature-2": {...}, "another-feature": {...}, } } }) ``` The data structure for "payload" is exactly the same as what is returned by the GrowthBook SDK endpoints and webhooks. You can update the payload at any time by calling `setPayload(newPayloadJSON)` and there are also `getPayload()` and `getDecryptedPayload()` methods, which are useful in hybrid apps where you want to hydrate the client with data from the server. Note: you don't need to specify `clientKey` or `apiHost` on your GrowthBook instance unless you want to enable streaming (see below) or call `refreshFeatures()` later. #### Synchronous Init There is a alternate synchronous version of init named `initSync`, which can be useful in some environments. There are some restrictions/differences: - You MUST pass in `payload` - The `payload` MUST NOT have encrypted features or experiments - If using sticky bucketing, you should use an instance of `StickyBucketServiceSync`, such as `BrowserCookieStickyBucketService`. - The return value is the GrowthBook instance to enable easy method chaining ## Streaming Updates The GrowthBook SDK supports streaming with Server-Sent Events (SSE). When enabled, changes to features within GrowthBook will be streamed to the SDK in realtime as they are published. This is only supported on GrowthBook Cloud or if running a GrowthBook Proxy Server. ### Streaming in Browser Environments SSE is supported on all major browsers, so enabling streaming is as easy as passing `streaming: true` into your `init` call: ```ts gb.init({ streaming: true, // Other settings... }) ``` You may also differentiate your streaming host URL from your API host by setting the `streamingHost` property in the GrowthBook constructor (ex: Remote Evaluation is done on a CDN edge worker while Streaming is done through a GrowthBook Proxy server). ## Remote Evaluation :::tip See the [Remote Evaluation overview](/self-host/remote-evaluation) for more information about what Remote Evaluation is, how it works, and deployment options. ::: When used in a front-end context, the JS SDK may be run in Remote Evaluation mode. This mode brings the security benefits of a backend SDK to the front end by evaluating feature flags exclusively on a private server. Using Remote Evaluation ensures that any sensitive information within targeting rules or unused feature variations are never seen by the client. Note that Remote Evaluation should not be used in a backend context. You must enable Remote Evaluation in your SDK Connection settings. Cloud customers are also required to self-host a GrowthBook Proxy Server, edge worker, or custom remote evaluation backend. To use Remote Evaluation, add the `remoteEval: true` property to your SDK instance. A new evaluation API call will be made any time a user attribute or other dependency changes. You may optionally limit these API calls to specific attribute changes by setting the `cacheKeyAttributes` property (an array of attribute names that, when changed, trigger a new evaluation call). ```ts const gb = new GrowthBook({ apiHost: "https://gb-proxy.mydomain.io/", clientKey: "sdk-abc123", // Enable remote evaluation remoteEval: true, // Optional: only trigger a new evaluation call when the `id` and `email` attribute changes cacheKeyAttributes: ["id", "email"], }); ``` :::note Sticky Bucketing with Remote Evaluation If you would like to implement Sticky Bucketing while using Remote Evaluation, you must configure your remote evaluation backend to support Sticky Bucketing. In the case of the GrowthBook Proxy Server, this means implementing a Redis database for sticky bucketing use. You will not need to provide a StickyBucketService instance to the client side SDK. ::: ## Caching The JavaScript SDK has 2 caching layers: 1. In-memory cache (available on all platforms) 2. Persistent localStorage cache (only available in browsers by default) There are a number of cache settings you can configure within GrowthBook. This must be done BEFORE creating a GrowthBook instance. Below are all of the default values. You can call `configureCache` with a subset of these fields and the rest will keep their default values. ```ts import { configureCache } from "@growthbook/growthbook"; configureCache({ // The localStorage key the cache will be stored under cacheKey: "gbFeaturesCache", // Consider features stale after this much time (60 seconds default) staleTTL: 1000 * 60, // Cached features older than this will be ignored (24 hours default) maxAge: 1000 * 60 * 60 * 24, // For Remote Eval only - limit the number of cache entries (~1 entry per user) maxEntries: 10, // When `false`, we add a `visibilitychange` listener to disable SSE when the page is idle disableIdleStreams: false, // Consider a page "idle" when it is hidden for this long (default 20 seconds) idleStreamInterval: 20000, // Set to `true` to completely disable both in-memory and persistent caching disableCache: false, }) ``` ## Polyfills The GrowthBook SDK relies on `fetch`, `SubtleCrypto`, `EventSource`, and `localStorage` being available at runtime. In browser environments these are natively available, but you can override any of them with `setPolyfills` if needed. This is especially useful when: - Running in a non-browser JS environment (e.g. testing with Jest or jsdom) where globals like `fetch` or `localStorage` may not be available - Using a custom `EventSource` implementation for streaming - Providing an alternative persistent storage backend in place of `localStorage` ```js import { setPolyfills } from "@growthbook/growthbook"; setPolyfills({ // Custom fetch implementation fetch: myCustomFetch, // Custom SubtleCrypto implementation SubtleCrypto: mySubtleCrypto, // Custom EventSource implementation (for streaming support) EventSource: myEventSource, // localStorage-compatible object localStorage: { getItem: (key) => myStorage.get(key), setItem: (key, value) => myStorage.set(key, value), }, }); ``` **Note**: `setPolyfills` must be called **before** creating a GrowthBook instance. For server-side (Node.js) usage, see the [Node.js SDK docs](/lib/node) which covers required polyfills for that environment. ## Re-rendering When Features Change When features change (e.g. by calling `gb.refreshFeatures()`), you need to re-render your app so that all of your feature flag checks can be re-evaluated. You can specify your own custom rendering function for this purpose: ```js // Callback to re-render your app when feature flag values change gb.setRenderer(() => { // TODO: re-render your app }); ``` ## Experimentation (A/B Testing) In order to run A/B tests, you need to set up a tracking callback function. This is called every time a user is put into an experiment and can be used to track the exposure event in your analytics system (Segment, Mixpanel, GA, etc.). **Note**: If you use the `third-party-tracking` plugin, you don't need to specify a `trackingCallback` since the plugin will handle that for you automatically. ```js const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", trackingCallback: (experiment, result, user) => { // Example using Segment analytics.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, attributes: user?.attributes, }); }, }); ``` Starting in version 1.7.0, the callback receives a third `user` argument: a `TrackingUserContext` containing the `attributes` and optional `url` used when the experiment was evaluated. We recommend recording attributes from this argument so exposure events consistently reflect the attributes GrowthBook used for targeting and assignment. This same tracking callback is used for both feature flag experiments and Visual Editor experiments. ### Feature Flag Experiments There is nothing special you have to do for feature flag experiments. Just evaluate the feature flag like you would normally do. If the user is put into an experiment as part of the feature flag, it will call the `trackingCallback` automatically in the background. ```js // If this has an active experiment and the user is included, // it will call trackingCallback automatically const newLogin = gb.isOn("new-signup-form"); ``` If the experiment came from a feature rule, `result.featureId` in the trackingCallback will contain the feature id, which may be useful for tracking/logging purposes. ### Visual Editor Experiments Experiments created through the GrowthBook Visual Editor will run automatically as soon as their targeting conditions are met. **Note**: Visual Editor experiments are only supported in a web browser environment. They will not run in Node.js, Mobile apps, or Desktop apps. If you are using this SDK in a Single Page App (SPA), you will need to let the GrowthBook instance know when the URL changes so the active experiments can update accordingly. ```js // Call this every time a navigation event happens in your SPA function onRouteChange() { gb.setURL(window.location.href); } ``` Visual Editor experiments are enabled by default, but can be disabled with various GrowthBook constructor settings: - **disableVisualExperiments** - If true, all visual editor experiments will be skipped - **disableJsInjection** - If true, any visual editor experiment that injects custom javascript will be skipped. #### Content Security Policy If you plan to use the Custom Javascript feature of the Visual Editor and you have a Content Security Policy on your site, there are two options: 1. Enable `unsafe-inline` script-src 2. OR generate a unique nonce value, add it to your script-src directive, and pass it into the GrowthBook constructor as `jsInjectionNonce` ### URL Redirect Experiments Similarly to Visual Editor experiments, URL redirect tests will run automatically if targeting conditions are met. If you are using this SDK in a Single Page App (SPA), you'll want to pass in a custom navigation function into the SDK (as default navigation for URL Redirects uses `window.location.replace(url)`) and set the `navigateDelay` to 0. ```js // Example in Next.js import router from "next/router"; const gb = new GrowthBook({ navigate: (url) => router.replace(url), navigateDelay: 0, // ... other settings }); ``` For SPA's you will also need to let the GrowthBook instance know when the URL changes so the active experiments can update accordingly. ```js // Call this every time a navigation event happens in your SPA function onRouteChange() { gb.setURL(window.location.href); } ``` URL Redirect experiments are enabled by default, but can be disabled with various GrowthBook constructor settings: - **disableUrlRedirectExperiments** - If true, all URL Redirect experiments will be skipped - **disableCrossOriginUrlRedirectExperiments** - If true, any URL Redirect with a destination pointing to a different origin will be skipped. ### Deferred Tracking Sometimes, your analytics tracker is loaded after GrowthBook. In that case, you should not specify a `trackingCallback` in the constructor and instead use `setTrackingCallback` later when ready. When you do this, the GrowthBook instance will queue up tracking calls and then fire them all at once when you set the callback. There are some scenarios where you need to queue up tracking calls in one GrowthBook instance and fire them in another. For example, if your analytics tracker is only available on the front-end, but you are running experiments in Node.js. Export the queued tracking calls with the `getDeferredTrackingCalls()` method. The result is a serializable JSON object containing the experiment, result, and `user` context captured during evaluation: ```ts const tracks = gb.getDeferredTrackingCalls(); ``` When these calls are serialized and sent between a server and client, the captured `user` context is passed as the third argument to the tracking callback. Deferred calls created by SDK versions earlier than 1.7.0 may not include `user`. Then, import with `setDeferredTrackingCalls`. This does not fire them automatically. You must call `fireDeferredTrackingCalls` after. ```ts gb2.setDeferredTrackingCalls(tracks); gb2.fireDeferredTrackingCalls(); ``` ### Sticky Bucketing Sticky bucketing ensures that users see the same experiment variant, even when user session, user login status, or experiment parameters change. See the [Sticky Bucketing docs](/app/sticky-bucketing) for more information. If your organization and experiment supports sticky bucketing, you must implement an instance of the `StickyBucketService` to use Sticky Bucketing. The JS SDK exports several implementations of this service for common use cases, or you may build your own: - `LocalStorageStickyBucketService` — For simple bucket persistence using the browser's LocalStorage (can be polyfilled for other environments). - `BrowserCookieStickyBucketService` — For simple bucket persistence using browser cookies, which are transportable to the back end. Assumes `js-cookie` is implemented (can be polyfilled). Cookie attributes can also be configured. The default cookie expiry is 180 days; override by passing `expires: {days}` into the constructor's `cookieAttributes`. - Build your own — Implement the abstract `StickyBucketService` class and connect to your own data store, or custom wrap multiple service implementations (ex: read/write to both cookies and Redis). Implementing most StickyBucketService implementations is straightforward and works with minimal setup. For instance, to use the `BrowserCookieStickyBucketService`: ```js import { BrowserCookieStickyBucketService } from "@growthbook/growthbook"; import Cookies from 'js-cookie'; const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", stickyBucketService: new BrowserCookieStickyBucketService({ jsCookie: Cookies, }), // ... }); ``` ## TypeScript When used in a TypeScript project, GrowthBook includes basic type inference out of the box: ```ts // Type will be `string` based on the fallback provided ("blue") const color = gb.getFeatureValue("button-color", "blue"); // You can manually specify types as well // feature.value will be type `number` const feature = gb.evalFeature("font-size"); console.log(feature.value); // Experiments will use the variations to infer the return value // result.value will be type "string" const result = gb.run({ key: "my-test", variations: ["blue", "green"], }); ``` ### Strict Typing If you want to enforce stricter types in your application, you can do that when creating the GrowthBook instance: ```ts // Define all your feature flags and types here interface AppFeatures { "button-color": string; "font-size": number; "newForm": boolean; } // Pass into the GrowthBook instance const gb = new GrowthBook({ ... }); ``` Now, all feature flag methods will be strictly typed. ```ts // feature.value will by type `number` const feature = gb.evalFeature("font-size"); console.log(feature.value); // Typos will cause compile-time errors gb.isOn("buton-color"); // "buton" instead of "button" ``` Instead of defining the `AppFeatures` interface manually like above, you can auto-generate it from your GrowthBook account using the [GrowthBook CLI](/tools/cli). ## Updating As a general philosophy, we aim to keep the SDK 100% backwards compatible at all times. View the [Changelog](https://github.com/growthbook/growthbook/blob/main/packages/sdk-js/CHANGELOG.md) for a complete list of all SDK changes. ### Updating to 1.0.0 Updating from a **0.X.X** release to **1.0.0** is still backwards compatible for the vast majority of use cases, although there are a few minor changes: - The `enableDevMode: true` setting previously also disabled cache as a side-effect. This is no longer the case in 1.0.0, and you must explicitly also set `disableCache: true` - Previously, a network request to fetch features was started immediately upon creating a GrowthBook instance. Starting in 1.0.0, it waits until you call `loadFeatures` (or the new `init` method) before starting the network request. As a replacement, there is now a standalone `prefetchPayload` function that you can use to kick off a network request outside of the context of a GrowthBook instance. #### Switching to init GrowthBook 1.0.0 introduced a new `init` (and `initSync`) method. We recommend everyone starts using this in their implementation. It solves many pain points including easier error handling and more control over caching and streaming. For those currently using `loadFeatures`, `init` is a direct replacement. The only difference is that streaming is now opt-in instead of on-by-default. ```ts // Previous await gb.loadFeatures({ timeout: 1000 }); // New await gb.init({ timeout: 1000, streaming: true }); ``` For those currently NOT using `loadFeatures` and passing features/experiments directly into the GrowthBook constructor, `init` and `initSync` can be used instead. The code below assumes you have a `payload` variable with the contents from the SDK Connection Endpoint. ```ts // Previous const gb = new GrowthBook({ features: payload.features, experiments: payload.experiments }); // New (async) const gb = new GrowthBook(); await gb.init({ payload: payload }); // New (non-async) const gb = (new GrowthBook()).initSync({ payload: payload }); ``` ## GrowthBook Instance (reference) ### Attributes You can specify attributes about the current user and request. These are used for two things: 1. Feature targeting (e.g. paid users get one value, free users get another) 2. Assigning persistent variations in A/B tests (e.g. user id "123" always gets variation B) The following are some commonly used attributes, but use whatever makes sense for your application. ```ts new GrowthBook({ attributes: { id: "123", loggedIn: true, deviceId: "abc123def456", company: "acme", paid: false, url: "/pricing", browser: "chrome", mobile: false, country: "US", }, }); ``` #### Updating Attributes If attributes change, you can call `setAttributes()` to update. This will completely overwrite any existing attributes. To do a partial update, use the following pattern: ```js gb.setAttributes({ // Only update the `url` attribute, keep the rest the same ...gb.getAttributes(), url: "/new-page" }) ``` #### Secure Attributes When _secure attribute hashing_ is enabled, all targeting conditions in the SDK payload referencing attributes with datatype `secureString` or `secureString[]` will be anonymized via SHA-256 hashing. This allows you to safely target users based on sensitive attributes. You must enable this feature in your SDK Connection for it to take effect. If your SDK Connection has secure attribute hashing enabled, you will need to manually hash any `secureString` or `secureString[]` attributes that you pass into the GrowthBook SDK. To hash an attribute, use a cryptographic library with SHA-256 support, and compute the SHA-256 hashed value of your attribute _plus_ your organization's secure attribute salt. The example below is using CryptoJS (https://www.npmjs.com/package/crypto-js), which provides a synchronous API. ```js import sha256 from 'crypto-js/sha256'; const salt = "f09jq3fij"; // Your secure attribute salt (set in Organization Settings) // hashing a secureString attribute const userEmail = sha256(salt + user.email).toString(); // hashing a secureString[] attribute const userTags = user.tags.map(tag => sha256(salt + tag).toString()); gb.setAttributes({ id: user.id, loggedIn: true, email: userEmail, tags: userTags, }); await gb.init(); ``` If you prefer to use a browser native implementation, you will need to await hashing before calling `setAttributes` since SubtleCrypto methods are async. Here's an example implementation: ```js // Native browser implementation async function sha256(str) { const buffer = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(str)); const hashArray = Array.from(new Uint8Array(buffer)); return hashArray.map(byte => byte.toString(16).padStart(2, "0")).join(""); } // Hashing a single string const userEmail = await sha256(salt + user.email); // Hashing an array of strings const userTags = await Promise.all( user.tags.map(tag => sha256(salt + tag)) ); ``` ### Feature Usage Callback GrowthBook can fire a callback whenever a feature is evaluated for a user. This can be useful to update 3rd party tools like NewRelic or DataDog. ```ts new GrowthBook({ onFeatureUsage: (featureKey, result) => { console.log("feature", featureKey, "has value", result.value); }, }); ``` The `result` argument is the same thing returned from `gb.evalFeature`. Note: If you evaluate the same feature multiple times (and the value doesn't change), the callback will only be fired the first time. ### Dev Mode There is a [GrowthBook Chrome DevTools Extension](https://chrome.google.com/webstore/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) that can help you debug and test your feature flags in development. In order for this to work, you must explicitly enable dev mode when creating your GrowthBook instance: ```js const gb = new GrowthBook({ enableDevMode: true, }); ``` To avoid exposing all of your internal feature flags and experiments to users, we recommend setting this to `false` in production in most cases. ### evalFeature In addition to the `isOn` and `getFeatureValue` helper methods, there is the `evalFeature` method that gives you more detailed information about why the value was assigned to the user. ```ts // Get detailed information about the feature evaluation const result = gb.evalFeature("my-feature"); // The value of the feature (or `null` if not defined) console.log(result.value); // Why the value was assigned to the user // One of: `override`, `unknownFeature`, `defaultValue`, `force`, or `experiment` console.log(result.source); // The string id of the rule (if any) which was used console.log(result.ruleId); // Information about the experiment (if any) which was used console.log(result.experiment); // The result of the experiment (or `undefined`) console.log(result.experimentResult); ``` ### Inline Experiments Instead of declaring all features up-front in the context and referencing them by ids in your code, you can also just run an experiment directly. This is done with the `gb.run` method: ```js // These are the only required options const { value } = gb.run({ key: "my-experiment", variations: ["red", "blue", "green"], }); ``` #### Customizing the Traffic Split By default, this will include all traffic and do an even split between all variations. There are 2 ways to customize this behavior: ```js // Option 1: Using weights and coverage gb.run({ key: "my-experiment", variations: ["red", "blue", "green"], // Only include 10% of traffic coverage: 0.1, // Split the included traffic 50/25/25 instead of the default 33/33/33 weights: [0.5, 0.25, 0.25], }); // Option 2: Specifying ranges gb.run({ key: "my-experiment", variations: ["red", "blue", "green"], // Identical to the above // 5% of traffic in A, 2.5% each in B and C ranges: [ [0, 0.05], [0.5, 0.525], [0.75, 0.775], ], }); ``` #### Hashing We use deterministic hashing to assign a variation to a user. We hash together the user's id and experiment key, which produces a number between `0` and `1`. Each variation is assigned a range of numbers, and whichever one the user's hash value falls into will be assigned. You can customize this hashing behavior: ```js gb.run({ key: "my-experiment", variations: ["A", "B"], // Which hashing algorithm to use // Version 2 is the latest and the one we recommend hashVersion: 2, // Use a different seed instead of the experiment key seed: "abcdef123456", // Use a different user attribute (default is `id`) hashAttribute: "device_id", }); ``` **Note**: For backwards compatibility, if no `hashVersion` is specified, it will fall back to using version `1`, which is deprecated. In the future, version `2` will become the default. We recommend specifying version `2` now for all new experiments to avoid migration issues down the line. #### Meta Info You can also define meta info for the experiment and/or variations. These do not affect the behavior, but they are passed through to the `trackingCallback`, so they can be used to annotate events. ```js gb.run({ key: "results-per-page", variations: [10, 20], // Experiment meta info name: "Results per Page", phase: "full-traffic" // Variation meta info meta: [ { key: "control", name: "10 Results per Page", }, { key: "variation", name: "20 Results per Page", }, ] }) ``` #### Mutual Exclusion Sometimes you want to run multiple conflicting experiments at the same time. You can use the `filters` setting to run mutually exclusive experiments. We do this using deterministic hashing to assign users a value between 0 and 1 for each filter. ```js // Will include 60% of users - ones with a hash between 0 and 0.6 gb.run({ key: "experiment-1", variation: [0, 1], filters: [ { seed: "pricing", attribute: "id", ranges: [[0, 0.6]] } ] }); // Will include the other 40% of users - ones with a hash between 0.6 and 1 gb.run({ key: "experiment-2", variation: [0, 1], filters: [ { seed: "pricing", attribute: "id", ranges: [[0.6, 1.0]] } ] }); ``` **Note** - If a user is excluded from an experiment due to a filter, the rule will be skipped and the next matching rule will be used instead. #### Holdout Groups To use global holdout groups, use a nested experiment design: ```js // The value will be `true` if in the holdout group, otherwise `false` const holdout = gb.run({ key: "holdout", variations: [true, false], // 10% of users in the holdout group weights: [0.1, 0.9] }); // Only run your main experiment if the user is NOT in the holdout if (!holdout.value) { const res = gb.run({ key: "my-experiment", variations: ["A", "B"] }) } ``` #### Targeting Conditions You can also define targeting conditions that limit which users are included in the experiment. These conditions are evaluated against the `attributes` passed into the GrowthBook context. The syntax for conditions is based on the MongoDB query syntax and is straightforward to read and write. For example, if the attributes are: ```json { "id": "123", "browser": { "vendor": "firefox", "version": 94 }, "country": "CA" } ``` The following condition would evaluate to `true` and the user would be included in the experiment: ```js gb.run({ key: "my-experiment", variation: [0, 1], condition: { "browser.vendor": "firefox", "country": { "$in": ["US", "CA", "IN"] } } }) ``` #### Inline Experiment Return Value A call to `gb.run(experiment)` returns an object with a few useful properties: ```ts const { value, key, name, variationId, inExperiment, hashUsed, hashAttribute, hashValue, } = gb.run({ key: "my-experiment", variations: ["A", "B"], }); // If user is included in the experiment console.log(inExperiment); // true or false // The index of the assigned variation console.log(variationId); // 0 or 1 // The value of the assigned variation console.log(value); // "A" or "B" // The key and name of the assigned variation (if specified in `meta`) console.log(key); // "0" or "1" console.log(name); // "" // If the variation was randomly assigned by hashing console.log(hashUsed); // The user attribute that was hashed console.log(hashAttribute); // "id" // The value of that attribute console.log(hashValue); // e.g. "123" ``` The `inExperiment` flag will be false if the user was excluded from being part of the experiment for any reason (e.g. failed targeting conditions). The `hashUsed` flag will only be true if the user was randomly assigned a variation. If the user was forced into a specific variation instead, this flag will be false. ## Feature Definitions (reference) The feature definition JSON file contains information about all of the features in your application. Each feature consists of a unique key, a list of possible values, and rules for how to assign those values to users. ```ts { "feature-1": {...}, "feature-2": {...}, "another-feature": {...}, } ``` ### Basic Feature An empty feature always has the value `null`: ```js { "my-feature": {} } ``` #### Default Values You can change the default assigned value with the `defaultValue` property: ```js { "my-feature": { defaultValue: "green" } } ``` ### Rules You can override the default value with **Rules**. Rules give you fine-grained control over how feature values are assigned to users. There are 2 types of feature rules: `force` and `experiment`. Force rules give the same value to everyone. Experiment rules assign values to users randomly. #### Rule Ids Rules can specify a unique identifier with the `id` property. This can help with debugging and QA by letting you see exactly why a specific value was assigned to a user. #### Rule Conditions Rules can optionally define targeting conditions that limit which users the rule applies to. These conditions are evaluated against the `attributes` passed into the GrowthBook context. The syntax for conditions is based on the MongoDB query syntax and is straightforward to read and write. For example, if the attributes are: ```json { "id": "123", "browser": { "vendor": "firefox", "version": 94 }, "country": "CA" } ``` The following condition would evaluate to `true`: ```json { "browser.vendor": "firefox", "country": { "$in": ["US", "CA", "IN"] } } ``` If a condition evaluates to `false`, the rule will be skipped. This means you can chain rules together with different conditions to support even the most complex use cases. #### Force Rules Force rules do what you'd expect - force a specific value for the feature ```js // Firefox users in the US or Canada get "green" // Everyone else gets the default "blue" { "button-color": { defaultValue: "blue", rules: [ { id: "rule-123", condition: { browser: "firefox", country: { $in: ["US", "CA"] } }, force: "green" } ], } } ``` ##### Gradual Rollouts You can specify a `range` for your rule, which determines what percent of users will get the rule applied to them. Users who do not get the rule applied will fall through to the next matching rule (or default value). You can also specify a `seed` that will be used for hashing. In order to figure out if a user is included or not, we use deterministic hashing. By default, we use the user attribute `id` for this, but you can override this by specifying `hashAttribute` for the rule: This is useful for gradually rolling out features to users (start with a small range and slowly increase). ```js { "new-feature": { defaultValue: false, rules: [ { force: true, hashAttribute: "device-id", seed: 'new-feature-rollout-abcdef123', // 20% of users range: [0, 0.2] // Increase to 40%: // range: [0, 0.4] } ] } } ``` #### Experiment Rules Experiment rules let you adjust the percent of users who get randomly assigned to each variation. This can either be used for hypothesis-driven A/B tests or to simply mitigate risk by gradually rolling out new features to your users. ```js // Each variation gets assigned to a random 1/3rd of users { "image-size": { rules: [ { variations: ["small", "medium", "large"] } ] } } ``` ##### Customizing the Traffic Split By default, an experiment rule will include all traffic and do an even split between all variations. There are 2 ways to customize this behavior: ```js // Option 1: Using weights and coverage { variations: ["red", "blue", "green"], // Only include 10% of traffic coverage: 0.1, // Split the included traffic 50/25/25 instead of the default 33/33/33 weights: [0.5, 0.25, 0.25] } // Option 2: Specifying ranges { variations: ["red", "blue", "green"], // Identical to the above // 5% of traffic in A, 2.5% each in B and C ranges: [ [0, 0.05], [0.5, 0.525], [0.75, 0.775] ] } ``` A user is assigned a number from 0 to 1 and whichever variation's range includes their number will be assigned to them. ##### Variation Meta Info You can use the `meta` setting to provide additional info about the variations such as name. ```js { "image-size": { rules: [ { variations: ["sm", "md", "lg"], ranges: [ [0, 0.5], [0.5, 0.75], [0.75, 1.0] ], meta: [ { key: "control", name: "Small", }, { key: "v1", name: "Medium", }, { key: "v2", name: "Large", } ] } ] } } ``` ##### Tracking Key and Name When a user is assigned a variation, we call the `trackingCallback` function so you can record the exposure with your analytics event tracking system. By default, we use the feature id to identify the experiment, but this can be overridden if needed with the `key` setting. You can also optionally provide a human-readable name. ```js { "feature-1": { rules: [ { // Use "my-experiment" as the key instead of "feature-1" key: "my-experiment", name: "My Experiment", variations: ["A", "B"] } ] }, } ``` ##### Hash Attribute We use deterministic hashing to make sure the same user always gets assigned the same value. By default, we use the attribute `id`, but this can be overridden with the `hashAttribute` setting: ```js const gb = new GrowthBook({ attributes: { id: "123", company: "acme", }, features: { "my-feature": { rules: [ // All users with the same "company" value // will be assigned the same variation { variations: ["A", "B"], hashAttribute: "company", }, // If "company" is empty for the user (e.g. if they are logged out) // The experiment will be skipped and fall through to this next rule { force: "A", }, ], }, }, }); ``` ##### Filters Sometimes you want to run multiple conflicting experiments at the same time. You can use the `filters` setting to run mutually exclusive experiments. We do this using deterministic hashing to assign users a value between 0 and 1 for each filter. ```js { "feature1": { rules: [ // Will include 60% of users - ones with a hash between 0 and 0.6 { variations: [false, true], filters: [ { seed: "pricing", attribute: "id", ranges: [[0, 0.6]] } ] } ] }, "feature2": { rules: [ // Will include the other 40% of users - ones with a hash between 0.6 and 1 { variations: [false, true], filters: [ { seed: "pricing", attribute: "id", ranges: [[0.6, 1.0]] } ] }, ] } } ``` **Note** - If a user is excluded from an experiment due to a filter, the rule will be skipped and the next matching rule will be used instead. ## Examples - [Typescript example app with strict typing ](https://github.com/growthbook/examples/tree/main/vanilla-typescript). ## Supported Features --- ## Kotlin (JVM) SDK import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' # Kotlin (JVM) This SDK supports Kotlin and Java backend applications running on the JVM, including server applications, CLI tools, and worker processes. It's optimized for server-side usage with suspend-first APIs and production-safe defaults. :::info Android Development? If you're building Android applications, see the [Kotlin (Android)](/lib/kotlin) documentation instead, which includes Android-specific guidance for Gradle Android plugin, Android engines, and AARs. ::: ## Supported Platforms - **JVM**: Java 8+ and Kotlin 1.5+ - **Server Applications**: Ktor, Spring Boot, Micronaut, Quarkus - **CLI Tools**: Command-line applications and scripts - **Worker Processes**: Background jobs and data processing - **Microservices**: Containerized applications and serverless functions ## Installation Add the GrowthBook JVM SDK and a network dispatcher to your project: ```kotlin dependencies { implementation("io.growthbook.sdk:GrowthBook-jvm:7.1.0") // Choose ONE network dispatcher (JVM-safe): implementation("io.growthbook.sdk:NetworkDispatcherOkHttp-jvm:1.0.7") // Recommended for servers // OR implementation("io.growthbook.sdk:NetworkDispatcherKtor-jvm:1.0.12") // If using Ktor CIO // Optional: JSON serialization helpers implementation("io.growthbook.sdk:GrowthBookKotlinxSerialization-jvm:1.0.0") } ``` ```groovy dependencies { implementation 'io.growthbook.sdk:GrowthBook-jvm:7.1.0' // Choose ONE network dispatcher (JVM-safe): implementation 'io.growthbook.sdk:NetworkDispatcherOkHttp-jvm:1.0.7' // Recommended for servers // OR implementation 'io.growthbook.sdk:NetworkDispatcherKtor-jvm:1.0.12' // If using Ktor CIO // Optional: JSON serialization helpers implementation 'io.growthbook.sdk:GrowthBookKotlinxSerialization-jvm:1.0.0' } ``` ```xml io.growthbook.sdk GrowthBook-jvm 7.1.0 io.growthbook.sdk NetworkDispatcherOkHttp-jvm 1.0.7 io.growthbook.sdk NetworkDispatcherKtor-jvm 1.0.12 io.growthbook.sdk GrowthBookKotlinxSerialization-jvm 1.0.0 ``` :::warning Important: Use JVM-specific Artifacts Always use the `-jvm` variants of the artifacts for server applications. Do not include Android artifacts or the Ktor Android engine, as they will add unnecessary dependencies and increase your application size. ::: ## Quick Start Here's a minimal example to get started with the GrowthBook Kotlin JVM SDK: ```kotlin import com.sdk.growthbook.GBSDKBuilder import com.sdk.growthbook.model.GBExperiment import com.sdk.growthbook.model.toGbString import com.sdk.growthbook.network.GBNetworkDispatcherOkHttp suspend fun main() { // User attributes for targeting and experiments val attributes = mapOf( "id" to "user_123".toGbString(), "environment" to "production".toGbString(), "organization" to "acme-corp".toGbString(), "role" to "admin".toGbString() ) // Create network dispatcher (OkHttp recommended for servers) val networkDispatcher = GBNetworkDispatcherOkHttp() // Build GrowthBook SDK instance val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = attributes, networkDispatcher = networkDispatcher, trackingCallback = { experiment, result -> // Track experiment views in your analytics println("Experiment: ${experiment.key}, Variation: ${result.variationId}") } ).initialize() // Fetch feature definitions growthBook.refreshCache() // Evaluate feature flags val newFeatureEnabled = growthBook.feature("new-checkout-flow").on val maxRetries = growthBook.featureValue("max-retries") ?: 3 println("New checkout flow: $newFeatureEnabled") println("Max retries: $maxRetries") // Run inline experiments val buttonColorExperiment = growthBook.run( GBExperiment( key = "button-color-test", variations = listOf("blue", "red", "green").map { it.toGbString() } ) ) println("Button color: ${buttonColorExperiment.value}") } ``` ## Network Dispatchers The GrowthBook SDK requires a network dispatcher for fetching feature definitions. Choose the appropriate one for your use case: ### OkHttp Dispatcher (Recommended) Best for most server applications due to its smaller dependency footprint: ```kotlin import com.sdk.growthbook.network.GBNetworkDispatcherOkHttp val networkDispatcher = GBNetworkDispatcherOkHttp( // Optional configuration client = customOkHttpClient, // Use your existing OkHttp client enableLogging = false, // Enable SDK-level request logging maxRetries = 10, // Max SSE reconnection retries initialRetryDelayMs = 1_000L, maxRetryDelayMs = 30_000L ) ``` ### Ktor CIO Dispatcher Use this if you're already using Ktor in your application: ```kotlin import com.sdk.growthbook.network.GBNetworkDispatcherKtor import io.ktor.client.HttpClient import io.ktor.client.engine.cio.* val networkDispatcher = GBNetworkDispatcherKtor( // Optional: supply a pre-configured Ktor HttpClient (CIO engine required for JVM) client = HttpClient(CIO) { // configure Ktor plugins here if needed }, enableLogging = false, maxRetries = 10, initialRetryDelayMs = 1_000L, maxRetryDelayMs = 30_000L ) ``` :::warning Avoid Android Engines When using Ktor, always specify the CIO engine explicitly. Do not use the Android engine in server applications as it will add unnecessary dependencies. ::: ## Configuration Configure the GrowthBook SDK for your server environment: ```kotlin val growthBook = GBSDKBuilder( // Required apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = userAttributes, networkDispatcher = networkDispatcher, // Optional configuration trackingCallback = { experiment, result -> // Your analytics tracking analyticsService.track("experiment_viewed", mapOf( "experiment_id" to experiment.key, "variation_id" to result.variationId, "user_id" to userAttributes["id"] )) }, // Enable local caching (enabled by default) cachingEnabled = true, // Enable debug logging to stdout enableLogging = false ) // Disable randomization for QA testing .setQAMode(false) // Callback fired when features are refreshed from the server .setRefreshHandler { success, error -> if (success) { logger.info("Features refreshed successfully") } else { logger.error("Feature refresh failed: ${error?.errorMessage}") } } .initialize() ``` ## Evaluating Features and Running Experiments ### Feature Flags Evaluate feature flags to control application behavior: ```kotlin // Simple boolean feature val newFeatureEnabled = growthBook.feature("new-checkout-flow").on if (newFeatureEnabled) { // Show new checkout flow showNewCheckoutFlow() } else { // Show legacy checkout showLegacyCheckout() } // Feature with default value val maxRetries = growthBook.featureValue("max-retries") ?: 3 val timeout = growthBook.featureValue("api-timeout") ?: 5000L // Complex feature values val config = growthBook.feature("service-config").gbValue as? GBJson val enabledServices = (config?.get("enabled") as? GBArray) ?.mapNotNull { (it as? GBString)?.value } ?: emptyList() ``` ### Feature Evaluation with Context Update user attributes for different contexts: ```kotlin // Evaluate for different users growthBook.setAttributes(mapOf( "id" to "user_123".toGbString(), "plan" to "premium".toGbString(), "country" to "US".toGbString() )) val premiumFeature = growthBook.feature("premium-dashboard").on // Switch to different user context growthBook.setAttributes(mapOf( "id" to "user_456".toGbString(), "plan" to "free".toGbString(), "country" to "CA".toGbString() )) val freeUserFeature = growthBook.feature("premium-dashboard").on // Will be different ``` ### Running Experiments Run A/B tests and experiments directly: ```kotlin // Simple A/B test val experiment = GBExperiment( key = "button-color-test", variations = listOf("blue", "red", "green").map { it.toGbString() } ).apply { weights = listOf(0.33f, 0.33f, 0.34f) } val result = growthBook.run(experiment) val buttonColor = result.value val inExperiment = result.inExperiment if (inExperiment) { println("User is in experiment, showing $buttonColor button") setButtonColor(buttonColor) } else { println("User not in experiment, using default") setButtonColor("blue") // default } // Complex experiment with targeting val pricingExperiment = GBExperiment( key = "pricing-test", variations = listOf(9.99, 14.99, 19.99).map { it.toGbNumber() } ).apply { weights = listOf(0.5f, 0.3f, 0.2f) condition = JsonObject(mapOf( "country" to JsonPrimitive("US"), "plan" to JsonPrimitive("premium") )) } val pricingResult = growthBook.run(pricingExperiment) if (pricingResult.inExperiment) { val price = pricingResult.value as GBNumber displayPrice(price) } ``` ### Experiment Tracking Track experiment exposures for analytics: ```kotlin val growthBook = GBSDKBuilder( // ... other config trackingCallback = { experiment, result -> // Send to your analytics platform analytics.track("experiment_viewed", mapOf( "experiment_id" to experiment.key, "variation_id" to result.variationId, "variation_value" to result.value, "user_id" to attributes["id"], "in_experiment" to result.inExperiment )) println("Tracked: ${experiment.key} -> ${result.value}") } ).initialize() ``` ### Advanced Features For more advanced usage including sticky bucketing, encrypted features, custom attributes, and mobile-specific considerations, see the [Kotlin (Android)](/lib/kotlin) documentation which covers these topics in detail. ## Serialization Support The optional serialization module provides helpers for working with `GBValue` and kotlinx.serialization: ```kotlin import com.sdk.growthbook.serialization.* @Serializable data class FeatureConfig( val maxRetries: Int, val timeout: Long, val enabledFeatures: List ) // Convert GBValue to typed objects val configValue = growthBook.feature("service-config").gbValue val config: FeatureConfig? = configValue.decodeAs() // Use the configuration config?.let { println("Max retries: ${it.maxRetries}") println("Timeout: ${it.timeout}") } ``` :::info When to Use Serialization You only need the serialization module if you want to work with complex JSON feature values as typed Kotlin objects. For simple boolean, string, and number features, you can skip this dependency. ::: ## Caching and Server-Sent Events ### Caching The builder exposes a `cachingEnabled` flag. On JVM, fetched features remain in the SDK instance after refresh, but there is no platform-provided persistent local cache layer: ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = attributes, networkDispatcher = networkDispatcher, trackingCallback = { _, _ -> }, cachingEnabled = true // true by default ).initialize() // Manually re-fetch and cache the latest features at any time growthBook.refreshCache() ``` To be notified when features are refreshed, use `setRefreshHandler()` on the builder: ```kotlin val builder = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = attributes, networkDispatcher = networkDispatcher, trackingCallback = { _, _ -> } ) builder.setRefreshHandler { success, error -> if (success) { logger.info("Features refreshed successfully") } else { logger.warn("Feature refresh failed: ${error?.errorMessage}") } } val growthBook = builder.initialize() ``` ### Real-time Updates with SSE Start a persistent Server-Sent Events connection for live feature updates: ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", streamingHost = "https://cdn.growthbook.io/", // SSE endpoint host attributes = attributes, networkDispatcher = networkDispatcher, trackingCallback = { _, _ -> } ).initialize() // Collect the SSE flow in a coroutine scope val flow = growthBook.startAutoRefreshFeatures() flow.launchIn(coroutineScope) // Stop the SSE connection when no longer needed growthBook.stopAutoRefreshFeatures() ``` ## Error Handling and Observability ### Exception Handling The SDK uses suspend functions and handles errors gracefully: ```kotlin try { growthBook.refreshCache() } catch (e: Exception) { logger.error("Failed to refresh features", e) // Use setRefreshHandler() to observe refresh success/failure callbacks } ``` ### Logging Integration The SDK outputs debug information to stdout when `enableLogging` is set to `true`. You can capture this in your application by redirecting stdout to your logging framework: ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = attributes, networkDispatcher = networkDispatcher, trackingCallback = { _, _ -> }, enableLogging = true // prints debug info to stdout ).initialize() ``` ## Testing ### Mock Dispatchers Use mock dispatchers for unit testing: ```kotlin import com.sdk.growthbook.network.NetworkDispatcher import com.sdk.growthbook.utils.Resource import com.sdk.growthbook.utils.SSEConnectionController import kotlinx.coroutines.Job import kotlinx.coroutines.flow.Flow import kotlinx.coroutines.flow.emptyFlow class MockNetworkDispatcher : NetworkDispatcher { var mockFeatures: String = """{"features": {}}""" override fun consumeGETRequest( request: String, onSuccess: (String) -> Unit, onError: (Throwable) -> Unit ): Job { onSuccess(mockFeatures) return Job() } override fun consumeSSEConnection( url: String, sseController: SSEConnectionController? ): Flow> = emptyFlow() override fun consumePOSTRequest( url: String, bodyParams: Map, onSuccess: (String) -> Unit, onError: (Throwable) -> Unit ) {} } // In your tests @Test fun testFeatureEvaluation() = runTest { val mockDispatcher = MockNetworkDispatcher() mockDispatcher.mockFeatures = """ { "features": { "test-feature": { "defaultValue": true, "rules": [] } } } """.trimIndent() val growthBook = GBSDKBuilder( apiKey = "test", apiHost = "http://localhost", attributes = mapOf("id" to "test-user".toGbString()), networkDispatcher = mockDispatcher, trackingCallback = { _, _ -> } ).initialize() growthBook.refreshCache() assertTrue(growthBook.feature("test-feature").on) } ``` ### Deterministic Testing Use local JSON fixtures for predictable tests: ```kotlin @Test fun testExperimentVariations() = runTest { val testFeatures = loadResourceAsString("/test-features.json") val mockDispatcher = MockNetworkDispatcher() mockDispatcher.mockFeatures = testFeatures val growthBook = GBSDKBuilder( apiKey = "test", apiHost = "http://localhost", attributes = mapOf("id" to "user_123".toGbString()), // Deterministic user networkDispatcher = mockDispatcher, trackingCallback = { _, _ -> } ).initialize() growthBook.refreshCache() // Test will always get the same variation for user_123 val result = growthBook.run( GBExperiment( key = "button-color-test", variations = listOf("blue", "red").map { it.toGbString() } ) ) assertEquals("blue", (result.value as GBString).value) // Deterministic based on user ID } ``` ## Performance and Resource Usage ### Connection Pooling Configure connection pooling for high-throughput applications: ```kotlin val customOkHttpClient = OkHttpClient.Builder() .connectionPool(ConnectionPool(10, 5, TimeUnit.MINUTES)) .connectTimeout(10, TimeUnit.SECONDS) .readTimeout(30, TimeUnit.SECONDS) .writeTimeout(10, TimeUnit.SECONDS) .build() val networkDispatcher = GBNetworkDispatcherOkHttp(client = customOkHttpClient) ``` ### Coroutine Dispatchers Use appropriate dispatchers for different workloads: ```kotlin import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.withContext // For CPU-intensive feature evaluation val result = withContext(Dispatchers.Default) { growthBook.feature("complex-feature").on } // For network operations (handled internally by SDK) val refreshed = withContext(Dispatchers.IO) { growthBook.refreshCache() } ``` ### Avoid Blocking Calls Never use `runBlocking` on request threads in server applications: ```kotlin // ❌ DON'T: Blocks the request thread fun handleRequest(request: HttpRequest): HttpResponse { val feature = runBlocking { growthBook.feature("new-feature").on } // ... } // ✅ DO: Use suspend functions end-to-end suspend fun handleRequest(request: HttpRequest): HttpResponse { val feature = growthBook.feature("new-feature").on // ... } ``` ## Framework Integration Examples ### Ktor Server ```kotlin import io.ktor.server.application.* import io.ktor.server.engine.* import io.ktor.server.netty.* import io.ktor.server.response.* import io.ktor.server.routing.* fun main() { embeddedServer(Netty, port = 8080) { // Initialize GrowthBook as singleton val growthBook = GBSDKBuilder( apiKey = System.getenv("GROWTHBOOK_API_KEY"), apiHost = "https://cdn.growthbook.io/", attributes = mapOf("environment" to "production".toGbString()), networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> } ).initialize() // Refresh features on startup launch { growthBook.refreshCache() } routing { get("/api/features/{userId}") { val userId = call.parameters["userId"] ?: return@get call.respond(400) // Create user-specific attributes val userAttributes = mapOf( "id" to userId.toGbString(), "environment" to "production".toGbString() ) // Update attributes for this request growthBook.setAttributes(userAttributes) // Evaluate features val features = mapOf( "newDashboard" to growthBook.feature("new-dashboard").on, "maxItems" to growthBook.featureValue("max-items"), "theme" to growthBook.featureValue("theme") ) call.respond(features) } } }.start(wait = true) } ``` ### Spring Boot ```kotlin import org.springframework.boot.autoconfigure.SpringBootApplication import org.springframework.boot.runApplication import org.springframework.context.annotation.Bean import org.springframework.context.annotation.Configuration import org.springframework.web.bind.annotation.* @SpringBootApplication class Application @Configuration class GrowthBookConfig { @Bean suspend fun growthBook(): GrowthBookSDK { val growthBook = GBSDKBuilder( apiKey = System.getenv("GROWTHBOOK_API_KEY"), apiHost = "https://cdn.growthbook.io/", attributes = mapOf("environment" to "production".toGbString()), networkDispatcher = GBNetworkDispatcherOkHttp(), trackingCallback = { _, _ -> } ).initialize() growthBook.refreshCache() return growthBook } } @RestController @RequestMapping("/api") class FeatureController(private val growthBook: GrowthBookSDK) { @GetMapping("/features/{userId}") suspend fun getUserFeatures(@PathVariable userId: String): Map { // Set user-specific attributes growthBook.setAttributes(mapOf( "id" to userId.toGbString(), "environment" to "production".toGbString() )) return mapOf( "newDashboard" to growthBook.feature("new-dashboard").on, "maxItems" to growthBook.featureValue("max-items"), "premiumFeatures" to growthBook.feature("premium-features").on ) } @PostMapping("/experiments/{userId}") suspend fun runExperiment( @PathVariable userId: String, @RequestParam experimentKey: String ): Map { growthBook.setAttributes(mapOf("id" to userId.toGbString())) val result = growthBook.run( GBExperiment( key = experimentKey, variations = listOf("control", "treatment").map { it.toGbString() } ) ) return mapOf( "variation" to result.value, "inExperiment" to result.inExperiment, "variationId" to result.variationId ) } } fun main(args: Array) { runApplication(*args) } ``` ### CLI Application ```kotlin import kotlinx.coroutines.runBlocking fun main(args: Array) = runBlocking { val userId = args.getOrNull(0) ?: "cli-user" // Initialize GrowthBook for CLI usage val growthBook = GBSDKBuilder( apiKey = System.getenv("GROWTHBOOK_API_KEY") ?: "sdk_dev_key", apiHost = "https://cdn.growthbook.io/", attributes = mapOf( "id" to userId.toGbString(), "environment" to "cli".toGbString(), "version" to "1.0.0".toGbString() ), networkDispatcher = GBNetworkDispatcherOkHttp(), trackingCallback = { _, _ -> } ).initialize() try { // Fetch latest features println("Fetching feature definitions...") growthBook.refreshCache() // Evaluate features for CLI behavior val verboseLogging = growthBook.feature("verbose-logging").on val maxConcurrency = growthBook.featureValue("max-concurrency") ?: 1 val outputFormat = growthBook.featureValue("output-format") ?: "json" println("Configuration:") println(" Verbose logging: $verboseLogging") println(" Max concurrency: $maxConcurrency") println(" Output format: $outputFormat") // Run your CLI logic here performCliTask(verboseLogging, maxConcurrency, outputFormat) } catch (e: Exception) { println("Error: ${e.message}") kotlin.system.exitProcess(1) } } suspend fun performCliTask(verbose: Boolean, concurrency: Int, format: String) { // Your CLI implementation if (verbose) { println("Running with verbose logging enabled") } println("Processing with concurrency level: $concurrency") println("Output format: $format") } ``` ## Versioning and Compatibility ### Module Versions GrowthBook Kotlin JVM modules follow independent versioning: | Module | Latest Version | Purpose | | ------------------------------------ | -------------- | ------------------- | | `GrowthBook-jvm` | 7.1.0 | Core SDK | | `NetworkDispatcherOkHttp-jvm` | 1.0.7 | OkHttp networking | | `NetworkDispatcherKtor-jvm` | 1.0.12 | Ktor CIO networking | | `GrowthBookKotlinxSerialization-jvm` | 1.0.0 | JSON serialization | ### Compatibility Matrix | SDK Version | Min Kotlin | Min JDK | OkHttp Dispatcher | Ktor Dispatcher | | ----------- | ---------- | ------- | ----------------- | --------------- | | 7.1.x | 1.5.0 | Java 8 | 1.0.7+ | 1.0.12+ | | 5.x.x | 1.4.0 | Java 8 | 1.0.1+ | 1.0.5+ | ### Dependency Management Pin JVM-specific versions explicitly in monorepos: ```kotlin // build.gradle.kts dependencies { // Explicitly specify JVM variants implementation("io.growthbook.sdk:GrowthBook-jvm:7.1.0") implementation("io.growthbook.sdk:NetworkDispatcherOkHttp-jvm:1.0.7") // Avoid mixing Android and JVM artifacts // ❌ Don't do this in server projects: // implementation("io.growthbook.sdk:GrowthBook:1.1.60") // Android variant } ``` ## Migration from Android Documentation If you started with the Android documentation but need server-side usage: ### Key Differences | Aspect | Android | JVM (Server) | | ------------- | ------------------------------------ | ---------------------------------------- | | **Artifacts** | `io.growthbook.sdk:GrowthBook:7.1.0` | `io.growthbook.sdk:GrowthBook-jvm:7.1.0` | | **Network** | Android engines OK | Use CIO or OkHttp only | | **Threading** | Main thread considerations | Suspend-first, no runBlocking | | **Lifecycle** | Activity/Fragment tied | Long-running singleton | | **Caching** | Persistent storage | No persistent local cache layer | ### Migration Steps 1. **Update Dependencies**: Replace Android artifacts with JVM variants 2. **Remove Android Engines**: Use OkHttp or Ktor CIO only 3. **Update Network Dispatcher**: Ensure JVM-compatible engines 4. **Review Threading**: Remove `runBlocking` from request handlers 5. **Update Caching**: Configure appropriate TTL for server usage ## Troubleshooting ### Common Issues **Problem**: `ClassNotFoundException` for Android classes ``` Solution: Ensure you're using -jvm artifacts, not Android variants ``` **Problem**: Large JAR size in server deployments ``` Solution: Use OkHttp dispatcher instead of Ktor with Android engine ``` **Problem**: Blocking network calls in request handlers ``` Solution: Use suspend functions end-to-end, avoid runBlocking ``` **Problem**: Features not updating in long-running processes ``` Solution: Enable SSE or implement periodic cache refresh ``` ### Debug Logging Enable debug logging to troubleshoot issues. The SDK prints to stdout when `enableLogging = true`: ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = attributes, networkDispatcher = networkDispatcher, trackingCallback = { _, _ -> }, enableLogging = true ).initialize() ``` ## Supported Features ## Further Reading - [Kotlin (Android) Documentation](/lib/kotlin) - For mobile app development - [Java SDK Documentation](/lib/java) - Alternative JVM SDK - [GrowthBook API Reference](/api) - REST API documentation - [Feature Flag Best Practices](/app/features) - Feature management guide --- **Need help?** Join our [Slack community](https://slack.growthbook.io) or check out our [GitHub repository](https://github.com/growthbook/growthbook-kotlin) for examples and support. --- ## Kotlin SDK import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' # Kotlin (Android) This SDK supports both Java and Kotlin Android apps using Android SDK 21 and above. :::info Server/Backend Development? If you're building JVM backend applications (servers, CLIs, workers), see the [Kotlin (JVM)](/lib/kotlin-jvm) documentation instead, which provides JVM-optimized guidance without Android-specific dependencies. ::: ## Installation ```kotlin repositories { mavenCentral() } dependencies { implementation("io.growthbook.sdk:GrowthBook:7.1.0") // Choose ONE network dispatcher: implementation("io.growthbook.sdk:NetworkDispatcherKtor:1.0.12") // OR implementation("io.growthbook.sdk:NetworkDispatcherOkHttp:1.0.7") // Optional: JSON serialization helpers implementation("io.growthbook.sdk:GrowthBookKotlinxSerialization:1.0.0") } ``` ```groovy repositories { mavenCentral() } dependencies { implementation 'io.growthbook.sdk:GrowthBook:7.1.0' // Choose ONE network dispatcher: implementation 'io.growthbook.sdk:NetworkDispatcherKtor:1.0.12' // OR implementation 'io.growthbook.sdk:NetworkDispatcherOkHttp:1.0.7' // Optional: JSON serialization helpers implementation 'io.growthbook.sdk:GrowthBookKotlinxSerialization:1.0.0' } ``` ### Network Dispatchers The SDK requires a network dispatcher for fetching feature definitions: - **`GBNetworkDispatcherKtor`** - Ktor-based implementation (Android-friendly default) - **`GBNetworkDispatcherOkHttp`** - OkHttp-based implementation (recommended for most Android apps) ## Quick Start To create a GrowthBook SDK instance, use `GBSDKBuilder`. The SDK uses coroutines, so initialize it from a coroutine scope. ```kotlin import com.sdk.growthbook.GBSDKBuilder import com.sdk.growthbook.GrowthBookSDK import com.sdk.growthbook.model.toGbBoolean import com.sdk.growthbook.model.toGbString import com.sdk.growthbook.network.GBNetworkDispatcherKtor import kotlinx.coroutines.launch class MainActivity : AppCompatActivity() { private lateinit var growthBook: GrowthBookSDK override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) // User attributes for targeting and assigning users to experiment variations val attributes = mapOf( "id" to "user-123".toGbString(), "deviceType" to "Android".toGbString(), "appVersion" to "1.2.0".toGbString(), "isPremium" to true.toGbBoolean() ) lifecycleScope.launch { growthBook = GBSDKBuilder( // Fetch and cache feature definitions from GrowthBook API apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = attributes, trackingCallback = { experiment, result -> // Track experiment views in your analytics system analytics.track("experiment_viewed", mapOf( "experiment_id" to experiment.key, "variation_id" to result.variationId )) }, networkDispatcher = GBNetworkDispatcherKtor(), // Optional: encryption key if using encrypted features encryptionKey = "your_encryption_key_here" ).initialize() } } } ``` ### Configuration Options The `GBSDKBuilder` accepts several configuration options: - **apiKey** (`String`, required) - Your GrowthBook API key - **apiHost** (`String`, required) - API host URL (typically `https://cdn.growthbook.io/`) - **streamingHost** (`String`, optional) - Streaming host URL for SSE updates - **attributes** (`Map`) - User attributes for targeting - **trackingCallback** (`(GBExperiment, GBExperimentResult) -> Unit`) - Analytics tracking callback - **networkDispatcher** (`NetworkDispatcher`, required) - Network implementation - **encryptionKey** (`String`, optional) - Decryption key for encrypted features - **cachingEnabled** (`Boolean`, default: true) - Enable/disable local feature caching - **remoteEval** (`Boolean`, default: false) - Enable remote feature evaluation - **enableLogging** (`Boolean`, default: false) - Print debug logs to stdout The following options are set via builder methods rather than constructor parameters: - `.setEnabled(Boolean)` - Enable/disable all experiments (default: true) - `.setQAMode(Boolean)` - Disable randomization for QA testing (default: false) - `.setForcedVariations(Map)` - Force specific variations for QA - `.setRefreshHandler(GBCacheRefreshHandler)` - Callback fired when features are refreshed - `.setStickyBucketService(GBStickyBucketService)` - Enable sticky bucketing - `.setFeatureUsageCallback(GBFeatureUsageCallback)` - Callback fired on every feature evaluation ### Updating User Attributes You can update user attributes at any time using `setAttributes()`. This completely replaces the attributes object: ```kotlin // Update attributes when user logs in growthBook.setAttributes(mapOf( "id" to userId.toGbString(), "isPremium" to true.toGbBoolean(), "country" to "US".toGbString(), )) ``` :::warning Attribute Changes Be aware that changing attributes may change the assigned feature values, which can be disorienting to users if not handled carefully. ::: ### Feature Refresh Handler To access features as soon as they're loaded from the backend, use `setRefreshHandler()`: ```kotlin class MainActivity : AppCompatActivity() { private var growthBookSDK: GrowthBookSDK? = null override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) lifecycleScope.launch { val builder = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = mapOf("id" to userId.toGbString()), networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> } ) builder.setRefreshHandler { isRefreshed, error -> if (isRefreshed) { // Features are now loaded and ready to use val feature = growthBookSDK?.feature("new-checkout-flow") if (feature?.on == true) { // Show new checkout flow } } else { Log.e("GrowthBook", "Failed to refresh features", error) } } growthBookSDK = builder.initialize() } } } ``` ## Evaluating Features ### Feature Result The `feature()` method takes a feature key and returns a `GBFeatureResult` object with the following properties: - **gbValue** (`GBValue`) - The assigned value of the feature (typed wrapper) - **on** (`Boolean`) - The value cast to a boolean - **off** (`Boolean`) - The value cast to a boolean and then negated - **source** (`GBFeatureSource`) - Why the value was assigned: `unknownFeature`, `defaultValue`, `force`, `experiment`, `prerequisite`, `cyclicPrerequisite`, or `override` When the source is `experiment`, there are additional properties: - **experiment** (`GBExperiment`) - The experiment configuration - **experimentResult** (`GBExperimentResult`) - The experiment evaluation result ### Basic Usage ```kotlin // Boolean feature flag val newCheckoutEnabled = growthBook.feature("new-checkout-flow").on if (newCheckoutEnabled) { showNewCheckout() } else { showLegacyCheckout() } // String feature val buttonColor = growthBook.feature("button-color").gbValue as? GBString // Numeric feature val maxRetries = growthBook.feature("max-retries").gbValue as? GBNumber // Complex JSON feature val config = growthBook.feature("service-config").gbValue as? GBJson ``` ### Working with GBValue Starting with version 2.0.0, feature values use the `GBValue` type for better type safety: ```kotlin val featureResult = growthBook.feature("my-feature") // Access the typed value when (val gbValue = featureResult.gbValue) { is GBBoolean -> { val boolValue = gbValue.value println("Boolean: $boolValue") } is GBString -> { val stringValue = gbValue.value println("String: $stringValue") } is GBNumber -> { val numberValue = gbValue.value // Double println("Number: $numberValue") } is GBArray -> { println("GbArray: $gbValue") } is GBJson -> { println("JSON: $gbValue") } is GBNull -> { println("Null value") } is GBValue.Unknown -> { println("GBValue.Unknown") } } ``` ### Typed Feature Access For convenience, you can directly get a typed feature value using `featureValue()` (available in version 7.1.0+): ```kotlin // Get feature value with inferred type val maxRetries: Int? = growthBook.featureValue("max-retries") val buttonColor: String? = growthBook.featureValue("button-color") val isEnabled: Boolean? = growthBook.featureValue("new-feature-enabled") // Use with default values val timeout = growthBook.featureValue("api-timeout") ?: 5000L ``` ### Feature Source and Experiments Check why a feature value was assigned and access experiment details: ```kotlin val feature = growthBook.feature("premium-feature") println("Feature value: ${feature.gbValue}") println("Is on: ${feature.on}") println("Source: ${feature.source}") // If the feature value came from an experiment if (feature.source == GBFeatureSource.experiment) { val experiment = feature.experiment val result = feature.experimentResult println("Experiment key: ${experiment?.key}") println("Variation ID: ${result?.variationId}") println("In experiment: ${result?.inExperiment}") } ``` ## Running Inline Experiments You can run experiments directly without defining them in the GrowthBook API. This is useful for programmatic experiments: ```kotlin val experiment = GBExperiment( key = "button-color-test" variations = listOf( "blue", "red", "green" ).map { it.toGbString() }, weights = listOf(0.5f, 0.3f, 0.2f), ) val result = growthBook.run(experiment) // Get the assigned variation val color = result.value as GBString // "blue", "red", or "green" println("Assigned color: $color") if (result.inExperiment) { // User is in the experiment applyButtonColor(color) } ``` ### Experiment Configuration The `GBExperiment` class accepts the following properties: **Required:** - **key** (`String`) - The unique identifier for this experiment - **variations** (`Array`) - Array of variations to choose between **Optional:** - **weights** (`FloatArray`) - Traffic distribution across variations (must sum to 1.0) - **active** (`Boolean`, default: true) - If false, always return control (first variation) - **coverage** (`Float`, default: 1.0) - Percentage of users to include (0.0 to 1.0) - **condition** (`GBCondition`) - Targeting conditions for the experiment - **namespace** (`GBNamespace`) - Namespace for experiment isolation - **force** (`Int`) - Force all users to a specific variation index (for QA) - **hashAttribute** (`String`, default: "id") - User attribute for variation assignment ### Experiment Result The `GBExperimentResult` object contains: - **inExperiment** (`Boolean`) - Whether the user is in the experiment - **variationId** (`Int`) - The index of the assigned variation - **value** (`Any`) - The value of the assigned variation - **hashAttribute** (`String`) - The attribute used for hashing - **hashValue** (`String`) - The value of the hash attribute - **key** (`String`) - The experiment key - **bucket** (`Float`) - The hash bucket value (0.0 to 1.0) - **stickyBucketUsed** (`Boolean`) - Whether sticky bucketing was used ### Advanced Experiment Example ```kotlin // Complex experiment with targeting and custom weights val pricingExperiment = GBExperiment( key = "pricing-test", variations = listOf(9.99, 14.99, 19.99) .map { it.toGbNumber() }, hashAttribute = "userId" // Use custom attribute for hashing ).apply { weights = listOf(0.5f, 0.3f, 0.2f) coverage = 0.8f // Only 80% of users // Only target premium users in the US condition = JsonObject(mapOf( "country" to JsonPrimitive("US"), "plan" to JsonPrimitive("premium"), )) } val result = growthBook.run(pricingExperiment) if (result.inExperiment) { val price = result.value as GBNumber displayPrice(price) // Track the experiment view analytics.track("pricing_experiment_viewed", mapOf( "price" to price, "variation_id" to result.variationId )) } ``` ## Sticky Bucketing Sticky Bucketing ensures users see consistent experiment variations even when targeting conditions or user attributes change. This prevents jarring user experiences from switching variations mid-experiment. ### How It Works When sticky bucketing is enabled, the SDK persists experiment assignments in local storage. The next time the user is evaluated for that experiment, they'll get the same variation they saw before. ### Implementation Implement the `GBStickyBucketService` interface to enable sticky bucketing: ```kotlin import com.sdk.growthbook.stickybucket.GBStickyBucketService import com.sdk.growthbook.utils.GBStickyAssignmentsDocument import android.content.SharedPreferences import kotlinx.coroutines.CoroutineScope import kotlinx.serialization.json.Json import kotlinx.serialization.encodeToString import kotlinx.serialization.decodeFromString class StickyBucketServiceImpl( override val coroutineScope: CoroutineScope, private val preferences: SharedPreferences, private val prefix: String = "gb_sticky_" ) : GBStickyBucketService { override suspend fun getAssignments( attributeName: String, attributeValue: String ): GBStickyAssignmentsDocument? { val key = "$prefix${attributeName}||$attributeValue" val json = preferences.getString(key, null) ?: return null return try { Json.decodeFromString(json) } catch (e: Exception) { null } } override suspend fun saveAssignments(doc: GBStickyAssignmentsDocument) { val key = "$prefix${doc.attributeName}||${doc.attributeValue}" val json = Json.encodeToString(doc) preferences.edit() .putString(key, json) .apply() } override suspend fun getAllAssignments( attributes: Map ): Map { val docs = mutableMapOf() attributes.forEach { (attrName, attrValue) -> getAssignments(attrName, attrValue)?.let { doc -> val docKey = "${doc.attributeName}||${doc.attributeValue}" docs[docKey] = doc } } return docs } } ``` ### Using Sticky Bucketing Pass your sticky bucket service implementation when building the SDK: ```kotlin val preferences = context.getSharedPreferences("growthbook_prefs", Context.MODE_PRIVATE) val stickyBucketService = StickyBucketServiceImpl( coroutineScope = lifecycleScope, preferences = preferences ) val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = mapOf("id" to userId.toGbString()), networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> } ) .setStickyBucketService(stickyBucketService) .initialize() ``` :::info Sticky Bucket Documents Each sticky bucket document contains: - `attributeName` - The attribute used to identify the user (e.g., "id", "deviceId") - `attributeValue` - The value of that attribute (e.g., "user-123") - `assignments` - A map of experiment assignments (e.g., `{"exp1__0": "control"}`) ::: ## Automatic Features Refresh The GrowthBook SDK supports automatic feature refresh through multiple mechanisms to ensure your app always has the latest feature definitions. ### Server-Sent Events (SSE) Server-Sent Events provide real-time feature updates without polling. When enabled, the SDK maintains a persistent connection to receive live updates whenever features change. ```kotlin val growthBook = gbSdkBuilder.initialize() val flow = growthBook.startAutoRefreshFeatures() flow.launchIn(lifecycleScope) ``` ### Manual Refresh You can manually trigger feature refresh at any time: ```kotlin lifecycleScope.launch { growthBook.refreshCache() Log.d("GrowthBook", "Feature refresh requested") } ``` ### Cache Configuration By default, caching is enabled. You can disable it via the `cachingEnabled` constructor parameter: ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = attributes, networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> }, cachingEnabled = false // disable local cache ).initialize() ``` To be notified when features are refreshed from the server, use `setRefreshHandler()` on the builder: ```kotlin val builder = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = attributes, networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> } ) builder.setRefreshHandler { isRefreshed, error -> if (isRefreshed) { Log.d("GrowthBook", "Features refreshed successfully") } else { Log.w("GrowthBook", "Feature refresh failed: ${error?.errorMessage}") } } val growthBook = builder.initialize() ``` ## Experiment Tracking and Feature Usage Callbacks The SDK provides comprehensive tracking capabilities for experiments and feature usage. ### Experiment Tracking Callback The tracking callback is called whenever a user is included in an experiment. This is essential for analytics and experiment analysis. ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = mapOf("id" to userId.toGbString()), networkDispatcher = GBNetworkDispatcherKtor(), // Experiment tracking callback trackingCallback = { experiment, result -> // Send to your analytics system analytics.track("experiment_viewed", mapOf( "experiment_id" to experiment.key, "variation_id" to result.variationId, "variation_value" to result.value, "user_id" to userId, "in_experiment" to result.inExperiment, "hash_used" to result.hashUsed )) // Optional: Track in your own analytics firebaseAnalytics.logEvent("experiment_viewed") { param("experiment_id", experiment.key) param("variation_id", result.variationId.toString()) } } ).initialize() ``` :::info Tracking Timing The tracking callback is only called when the user is actually included in the experiment. If they're not in the experiment, this callback won't be triggered. ::: ### Feature Usage Callback The feature usage callback is called every time a feature is evaluated, regardless of whether it's part of an experiment or not. ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = mapOf("id" to userId.toGbString()), networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> } ) .setFeatureUsageCallback { featureKey, result -> // Track feature usage for monitoring and debugging Log.d("GrowthBook", "Feature evaluated: $featureKey = ${result.gbValue}") // Send to monitoring systems monitoring.trackFeatureUsage(featureKey, result.gbValue) // Optional: Track in analytics analytics.track("feature_used", mapOf( "feature_key" to featureKey, "feature_value" to result.gbValue.toString(), "source" to result.source.name, "user_id" to userId )) } .initialize() ``` ### Advanced Tracking Example Combine both callbacks for comprehensive tracking: ```kotlin class GrowthBookManager { private val analytics: AnalyticsService private val monitoring: MonitoringService fun initializeGrowthBook(userId: String): GrowthBookSDK { return GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = mapOf("id" to userId.toGbString()), networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { experiment, result -> trackExperiment(experiment, result, userId) } ) .setFeatureUsageCallback { featureKey, result -> trackFeatureUsage(featureKey, result, userId) } .initialize() } private fun trackExperiment(experiment: GBExperiment, result: GBExperimentResult, userId: String) { // Send to multiple analytics platforms analytics.track("experiment_viewed", mapOf( "experiment_id" to experiment.key, "variation_id" to result.variationId, "user_id" to userId )) // Track in monitoring for alerting monitoring.recordExperimentView(experiment.key, result.variationId) } private fun trackFeatureUsage(featureKey: String, result: GBFeatureResult, userId: String) { // Monitor feature usage patterns monitoring.recordFeatureUsage(featureKey, result.gbValue) // Track feature performance if (result.source == GBFeatureSource.experiment) { analytics.track("feature_experiment_used", mapOf( "feature_key" to featureKey, "experiment_id" to result.experiment?.key, "user_id" to userId )) } } } ``` ## Encrypted Features and Secure Attributes GrowthBook supports encryption for sensitive feature definitions and secure attribute hashing for privacy protection. ### Encrypted Features Encrypted features ensure that sensitive feature configurations never reach the client in plain text. #### Setup Encrypted Features 1. **Enable encryption in GrowthBook**: Go to your SDK Connection settings and enable "Encrypt SDK Payload" 2. **Get your encryption key**: Copy the encryption key from the SDK Connection settings 3. **Configure the SDK**: ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = mapOf("id" to userId.toGbString()), networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> }, // Provide your encryption key encryptionKey = "your_encryption_key_here" ).initialize() ``` #### Working with Encrypted Features ```kotlin // The SDK automatically decrypts features when evaluating val encryptedFeature = growthBook.feature("sensitive-feature") val value = encryptedFeature.gbValue // You can check if encryption is working by looking at the source if (encryptedFeature.source == GBFeatureSource.defaultValue) { Log.d("GrowthBook", "Feature decrypted successfully") } ``` :::warning Security Best Practices - Store encryption keys securely (use Android Keystore or environment variables) - Never commit encryption keys to version control - Rotate encryption keys regularly - Use different keys for different environments ::: ### Secure Attributes Secure attributes allow you to target users based on sensitive information without exposing that information to the client. #### Setup Secure Attributes 1. **Enable secure attribute hashing** in your SDK Connection settings 2. **Hash sensitive attributes** before passing them to the SDK: ```kotlin import java.security.MessageDigest import java.nio.charset.StandardCharsets class SecureAttributeManager { private val salt = "your_organization_salt" // From Organization Settings fun hashSecureAttribute(value: String): String { val input = value + salt val digest = MessageDigest.getInstance("SHA-256") val hash = digest.digest(input.toByteArray(StandardCharsets.UTF_8)) return hash.joinToString("") { "%02x".format(it) } } fun createSecureAttributes(user: User): Map { return mapOf( "id" to user.id.toGbString(), "email" to hashSecureAttribute(user.email).toGbString(), // Hash sensitive email "phone" to hashSecureAttribute(user.phone).toGbString(), // Hash sensitive phone "country" to user.country.toGbString(), // Non-sensitive attributes remain plain "plan" to user.plan.toGbString() ) } } ``` #### Using Secure Attributes ```kotlin val secureManager = SecureAttributeManager() // Hash sensitive attributes before setting them val secureAttributes = secureManager.createSecureAttributes(currentUser) val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = secureAttributes, networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> } ).initialize() // The SDK will use hashed attributes for targeting val targetedFeature = growthBook.feature("premium-feature") ``` #### Advanced Secure Attribute Example ```kotlin class UserAttributeManager { private val salt = BuildConfig.GROWTHBOOK_SECURE_ATTRIBUTE_SALT private fun sha256(input: String): String { val digest = MessageDigest.getInstance("SHA-256") val hash = digest.digest(input.toByteArray(StandardCharsets.UTF_8)) return hash.joinToString("") { "%02x".format(it) } } fun buildUserAttributes(user: User): Map { val attributes = mutableMapOf() // Non-sensitive attributes attributes["id"] = user.id.toGbString() attributes["country"] = user.country.toGbString() attributes["plan"] = user.plan.toGbString() attributes["signupDate"] = user.signupDate.toGbString() // Hash sensitive attributes if (user.email.isNotEmpty()) { attributes["email"] = sha256(user.email + salt).toGbString() } if (user.phone.isNotEmpty()) { attributes["phone"] = sha256(user.phone + salt).toGbString() } // Hash PII for targeting attributes["userSegment"] = sha256("${user.id}_${user.plan}" + salt).toGbString() return attributes } } // Usage in your app val attributeManager = UserAttributeManager() val attributes = attributeManager.buildUserAttributes(currentUser) growthBook.setAttributes(attributes) ``` ### Custom Attributes You can define custom attributes for advanced targeting scenarios: ```kotlin import com.sdk.growthbook.model.GBArray val customAttributes = mapOf( // Standard attributes "id" to userId.toGbString(), "country" to "US".toGbString(), // Custom business attributes "lifetimeValue" to userLifetimeValue.toGbNumber(), "lastPurchaseDate" to lastPurchaseTimestamp.toGbString(), "preferredCategory" to userPreferences.category.toGbString(), "deviceModel" to Build.MODEL.toGbString(), "appVersion" to BuildConfig.VERSION_NAME.toGbString(), // Computed attributes "isHighValueCustomer" to (userLifetimeValue > 1000).toGbBoolean(), "daysSinceLastPurchase" to daysSinceLastPurchase.toGbNumber(), // Array attributes for complex targeting "purchasedCategories" to GBArray(listOf("electronics", "books", "clothing").map { it.toGbString() }), "subscriptionFeatures" to GBArray(userSubscriptions.features.map { it.toGbString() }) ) val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = customAttributes, networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> } ).initialize() ``` ## Remote Evaluation :::tip See the [Remote Evaluation overview](/self-host/remote-evaluation) for more information about what Remote Evaluation is, how it works, and deployment options. ::: Remote Evaluation evaluates feature flags on a secure server instead of the client, ensuring sensitive targeting rules and unused variations never reach the client. ### When to Use Use Remote Evaluation when you need to: - Keep targeting rules private - Hide unused feature variations - Prevent users from seeing all possible values - Add an extra layer of security :::warning Backend Required Remote Evaluation requires either: - GrowthBook Cloud with Remote Evaluation enabled - A self-hosted GrowthBook Proxy Server - A custom remote evaluation backend ::: ### Setup Enable Remote Evaluation in your SDK Connection settings in GrowthBook, then configure the SDK: ```kotlin val growthBook = GBSDKBuilder( apiKey = "sdk_abc123", apiHost = "https://cdn.growthbook.io/", attributes = mapOf("id" to userId.toGbString()), networkDispatcher = GBNetworkDispatcherKtor(), trackingCallback = { _, _ -> }, remoteEval = true // Enable remote evaluation ).initialize() ``` When Remote Evaluation is enabled, the SDK will make an API call to evaluate features whenever user attributes change. :::info Sticky Bucketing with Remote Evaluation If using Sticky Bucketing with Remote Evaluation, configure sticky bucketing on your remote evaluation backend. You don't need to provide a `StickyBucketService` to the client SDK. ::: ## Serialization Support The optional `GrowthBookKotlinxSerialization` module provides helpers for working with complex feature values using kotlinx.serialization. ### Installation ```kotlin dependencies { implementation("io.growthbook.sdk:GrowthBookKotlinxSerialization:1.0.0") } ``` ### Usage Define your data classes and use the serialization helpers: ```kotlin import kotlinx.serialization.Serializable import com.sdk.growthbook.serialization.* @Serializable data class AppConfig( val apiTimeout: Long, val maxRetries: Int, val enabledFeatures: List, val themeColors: Map ) // Get a complex feature value as a typed object val configFeature = growthBook.feature("app-config") val config: AppConfig? = configFeature.gbValue.decodeAs() config?.let { println("API Timeout: ${it.apiTimeout}") println("Max Retries: ${it.maxRetries}") println("Enabled Features: ${it.enabledFeatures.joinToString()}") // Use the configuration in your app setupApiClient(timeout = it.apiTimeout, retries = it.maxRetries) } ``` :::info When to Use You only need the serialization module if you work with complex JSON feature values and want type-safe deserialization. For simple boolean, string, and number features, you can skip this dependency. ::: ## ProGuard Configuration (Android) If you use ProGuard or R8 for code shrinking and obfuscation, add these rules to your `proguard-rules.pro`: ```proguard # Core GrowthBook SDK -keep class com.sdk.growthbook.** { *; } # Kotlinx Serialization -keep class kotlinx.serialization.json.** { *; } -keepattributes *Annotation*, InnerClasses -dontnote kotlinx.serialization.SerializationKt # Keep serializers -keep,includedescriptorclasses class com.sdk.growthbook.**$$serializer { *; } # Keep companion objects -keepclassmembers class com.sdk.growthbook.** { *** Companion; } # Keep classes with KSerializer -keepclasseswithmembers class com.sdk.growthbook.** { kotlinx.serialization.KSerializer serializer(...); } ``` :::warning Important These are baseline rules. Depending on your app's configuration and the features you use, you may need additional rules. Test thoroughly with ProGuard enabled. ::: ## Version History and Breaking Changes The Kotlin SDK has undergone several major version updates with breaking changes: ### Recent Versions - **v1.1.63** (2024-11-26) - Changed `value` field type to `kotlinx.serialization.json.JsonElement` - **v2.0.0** (2025-01-10) - Renamed `value` to `gbValue` with `GBValue` type; added typed `feature()` method - **v3.0.0** (2025-01-27) - Changed user attributes to use `GBValue` types - **v4.0.0** (2025-03-03) - Changed `initialize()` to suspend method - **v5.0.0** (2025-05-22) - Moved GBValue to Core module - **v6.0.0** (2025-05-22) - Renamed `hostURL` to `apiHost`, added `streamingHost` - **v6.1.0** (2025-08-15) - Changed `GBStickyBucketService` methods to suspend, added `coroutineScope` - **v7.1.0** (2026-04-07) - Added `featureValue()` and hid reified typed APIs from Objective-C ### Migration Guidance When upgrading between major versions, review the changelog on the [GitHub repository](https://github.com/growthbook/growthbook-kotlin) for detailed migration instructions. ## Further Reading The GitHub repository contains comprehensive documentation including: - Detailed API reference - Advanced usage examples - Integration guides - Contributing guidelines **GitHub Repository:** https://github.com/growthbook/growthbook-kotlin ## Supported Features --- ## Next.js SDK (Flags) import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Next.js SDK (Vercel Flags) This documentation is intended for implementing the GrowthBook adapter for Vercel's [Flags SDK](https://flags-sdk.dev/) in Next.js back-end or hybrid environments. For front-end-only GrowthBook implementations, please see our [React docs](/lib/react). ## Installation Install with a package manager ```bash npm2yarn npm install --save @flags-sdk/growthbook ``` ## Adapter Usage ### Import the Default Adapter A default adapter is available for use, assuming the appropriate environment variables are set ```ts import { growthbookAdapter } from '@flags-sdk/growthbook'; ``` #### Environment Variables The default adapter automatically uses following environment variables: ```bash # Required GROWTHBOOK_CLIENT_KEY="sdk-abc123" # Optional for cloud customers, required for self-hosted GROWTHBOOK_API_HOST="https://api.your-growthbook-instance.io" # Flags Explorer (Optional) GROWTHBOOK_API_KEY="secret_user_abcd1234" # Self-hosted only GROWTHBOOK_APP_ORIGIN="https://app.your-growthbook-instance.io" # Edge Config (Optional) EXPERIMENTATION_CONFIG="your-vercel-edge-config" # for native Vercel integrations # or GROWTHBOOK_EDGE_CONNECTION_STRING="your-edge-connection-string" # Optional: Edge Config is keyed by `clientKey` by default. To override, set: GROWTHBOOK_EDGE_CONFIG_ITEM_KEY="gb_payload" ``` ### Create a Custom Adapter You can provide custom configuration by using `createGrowthBookAdapter`: ```ts import { createGrowthBookAdapter } from '@flags-sdk/growthbook'; const myGrowthBookAdapter = createGrowthBookAdapter({ clientKey: process.env.GROWTHBOOK_CLIENT_KEY!, apiHost: process.env.GROWTHBOOK_API_HOST, // optional appOrigin: process.env.GROWTHBOOK_APP_ORIGIN, // optional edgeConfig: { connectionString: process.env.GROWTHBOOK_EDGE_CONNECTION_STRING!, itemKey: process.env.GROWTHBOOK_EDGE_CONFIG_ITEM_KEY, // optional }, trackingCallback: (experiment, result) => { // Back-end exposure logging }, clientOptions: {}, // GrowthBook ClientOptions (optional) initOptions: {}, // GrowthBook InitOptions (optional) stickyBucketService: undefined, // Optional }); ``` ## User Identification GrowthBook uses Attributes to evaluate feature flags and experiments. You should write an identify function providing these Attributes to GrowthBook flags: ```ts import { dedupe, flag } from 'flags/next'; import type { Identify } from 'flags'; import { growthbookAdapter, type Attributes } from '@flags-sdk/growthbook'; const identify = dedupe((async ({ headers, cookies }) => { return { id: cookies.get('user_id')?.value, // etc... }; }) satisfies Identify); export const myFeatureFlag = flag({ key: 'my_feature_flag', identify, adapter: growthbookAdapter.feature(), }); ``` **Dedupe** is used above to ensure that the Attributes are computed once per request. ## Adapter Methods and Properties ### `.feature()` This method implements the Adapter interface for a GrowthBook feature. Typically flag definitions are applied in single file (e.g. `flags.ts`). ```ts export const myFlag = flag({ key: 'my_flag', adapter: growthbookAdapter.feature(), defaultValue: "default", identify, }); ``` You may optionally pass in an options object for further customization (ex: `feature({ exposureLogging: false })`) | Option | Default | Description | | --------------- | ------- | -------------------------------- | | exposureLogging | true | Enable/disable exposure logging. | ### `.initialize()` Initializes the GrowthBook SDK. This is done on-demand when a growthbook flag is evaluated, and is not required to be called manually: ```ts const growthbookClient = await growthbookAdapter.initialize(); ``` ### `.setTrackingCallback()` Set a back-end callback to handle experiment exposures. This allows you to log exposures to your analytics platform. Typically this is done in the same file where your flags are defined (e.g. `flags.ts`). ```ts import { growthbookAdapter } from '@flags-sdk/growthbook'; import { after } from 'next/server'; growthbookAdapter.setTrackingCallback((experiment, result) => { // Safely fire and forget async calls (Next.js) after(async () => { console.log('Viewed Experiment', { experimentId: experiment.key, variationId: result.key, }); }); }); ``` Front-end experiment tracking is also supported, although it requires additional manual setup. See [client-side tracking](#client-side-tracking) for more information. ### `.setStickyBucketService()` Sticky bucketing ensures that users see the same experiment variant, even when user session, user login status, or experiment parameters change. See the [Sticky Bucketing docs](/app/sticky-bucketing) for more information. If your organization and experiment supports sticky bucketing, you must implement an instance of the `StickyBucketService` to use Sticky Bucketing. ```ts import { growthbookAdapter } from '@flags-sdk/growthbook'; import { RedisStickyBucketService } from '@growthbook/growthbook'; import Redis from 'ioredis'; const redis = new Redis(process.env.REDIS_CONNECTION_URL); const redisStickyBucketService = new RedisStickyBucketService({ redis }); growthbookAdapter.setStickyBucketService(redisStickyBucketService); ``` ### `.growthbook` You may access the underlying GrowthBook instance. Specifically, our Flags SDK adapter wraps the `GrowthBookClient` class [read more](/lib/js#). All user evaluation options (attributes, tracking callbacks, sticky buckets) are applied at the `userContext` level (not globally). ### `.stickyBucketService` If you have set a sticky bucket service, you may retrieve its instance. ## Edge Config The adapter can load your SDK payload from Vercel's Edge Config to lower the latency of feature flag evaluation: - Set `GROWTHBOOK_EDGE_CONNECTION_STRING` (or `EXPERIMENTATION_CONFIG` if installed through the Vercel Marketplace) in your environment. Optionally set `GROWTHBOOK_EDGE_CONFIG_ITEM_KEY` to override the default key name (defaults to your client key). - Or pass `edgeConfig` directly to the adapter. If Edge Config is not set, the adapter will fetch configuration from GrowthBook's API. ### Configuring a SDK Webhook 1. To automatically populate the Edge Config whenever your feature definitions change, create a [SDK Webhook](/app/webhooks/sdk-webhooks) on the same SDK Connection that you are using for the Next.js integration.ts 2. Select "Vercel Edge Config" as the webhook type and fill out the following fields: - Vercel Edge Config ID (begins with `ecfg_`) - Team ID (optional) - Vercel API Token (see Vercel → Account Settings → Tokens) Under the hood, the webhook is being configured with the following properties. If you need to change any of these settings for any reason, you can always edit the webhook. - **Endpoint URL** is being set to ``` https://api.vercel.com/v1/edge-config/{edge_config_id}/items ``` - **Method** is being set to `PATCH` - An **Authorization: Bearer token** header is being added with your Vercel API Token - The **Payload format** is being set to `Vercel Edge Config` :::note Vercel Edge Config limitations Vercel's Edge Config is subject to storage size limitations. If your SDK payload is excessively large, you may not be able to populate the your Edge Config. Read more about limitations [here](https://vercel.com/docs/edge-config/edge-config-limits). ::: ## Additional Configuration - **Initialization:** The adapter auto-initializes when a flag is evaluated. To pre-initialize, call `initialize()` manually. - **Exposure Logging:** By default, exposures are logged when flags are evaluated. You can disable this with `exposureLogging: false` or provide a custom tracking callback (applicable to back-end tracking only). ## Flags Explorer Integration To expose GrowthBook data to the Flags Explorer, use the `getProviderData` function in your API route: ```ts import { getProviderData, createFlagsDiscoveryEndpoint } from 'flags/next'; import { getProviderData as getGrowthBookProviderData } from '@flags-sdk/growthbook'; import { mergeProviderData } from 'flags'; import * as flags from '../../../../flags'; export const GET = createFlagsDiscoveryEndpoint(async (request) => { return mergeProviderData([ getProviderData(flags), getGrowthBookProviderData({ apiKey: process.env.GROWTHBOOK_API_KEY, clientKey: process.env.GROWTHBOOK_CLIENT_KEY, /** Self-hosted users should provide additional options: **/ // appApiHost: process.env.GROWTHBOOK_API_HOST, // appOrigin: process.env.GROWTHBOOK_APP_ORIGIN, }), ]); }); ``` Note the required environment variable `GROWTHBOOK_API_KEY`. You will need to create and provide either a Personal Access Token (read-only) or an API Key. You will also need to provide a `FLAGS_SECRET` environment variable. See [Vercel's Flags Explorer guide](https://vercel.com/docs/feature-flags/flags-explorer/reference#flags_secret-environment-variable). ## Client-side Tracking When using the Flags SDK, we strongly encourage using server-side tracking when possible. However, some event trackers are best suited for client-side implementation. Additionally, some user attributes may only be available in a front-end context. We recommend implementing client-side tracking using some glue components: 1. A `` server component which takes a list of evaluated feature ids and prepares the data necessary to hydrate the tracking calls for the client. It embeds a `` client component. 2. A `` client component responsible for executing and deduping your tracking calls in the browser. You will need to define a client-side tracking callback within this component. Under the hood, this component works by replaying all of the experiment evaluations in a client context. 3. You must include `` in any component which evaluates a feature flag (or specifically: all referenced flags that may trigger experiment tracking callbacks). **1. <GrowthbookTracking>** `components/growthbook/client-side-tracking/growthbook-tracking.tsx` ```tsx import { growthbookAdapter, type Attributes, type StickyAssignmentsDocument, } from '@flags-sdk/growthbook'; import { identify } from '@/lib/identify'; import { GrowthbookTrackingClient } from './client'; export async function GrowthbookTracking({ featureIds }: { featureIds: string[] }) { // The GrowthBook SDK must be ready before tracking await growthbookAdapter.initialize(); const payload = growthbookAdapter.growthbook.getDecryptedPayload(); const attributes: Attributes = await identify(); // If using sticky bucketing, extract the docs for client side hydration let stickyBucketAssignmentDocs: Record | undefined; if (growthbookAdapter.stickyBucketService) { const ctx = await growthbookAdapter.growthbook.applyStickyBuckets( { attributes }, growthbookAdapter.stickyBucketService ); stickyBucketAssignmentDocs = ctx.stickyBucketAssignmentDocs; } return ( ); } ``` **2. <GrowthbookTrackingClient>** `components/growthbook/client-side-tracking/client.tsx` ```tsx 'use client'; import { GrowthBookClient, type Attributes, type TrackingCallback, type StickyAssignmentsDocument, type FeatureApiResponse, type UserContext, } from '@growthbook/growthbook'; import { useEffect } from 'react'; // Define your client-side tracking callback: // You may also use type TrackingCallbackWithUser if your tracking library requires user context. const trackingCallback: TrackingCallback = (experiment, result) => { console.log('Viewed Experiment (client-side tracking)', { experimentId: experiment.key, variationId: result.key, }); } // Create a client-side JS SDK to replay the experiments: // If using TrackingCallbackWithUser, pass the tracking callback into GrowthBookClient constructor instead of evalFeature. const growthbook = new GrowthBookClient(); // Dedupe tracked experiments (evalFeature automatically mutates this): const trackedExperiments = new Set() export function GrowthbookTrackingClient({ featureIds, attributes, payload, stickyBucketAssignmentDocs, }: { featureIds: string[]; attributes: Attributes; payload: FeatureApiResponse; stickyBucketAssignmentDocs?: Record; }) { useEffect(() => { growthbook.initSync({ payload }); // saveStickyBucketAssignmentDoc is a no-op because SBs are persisted server-side featureIds.forEach((fid) => { const ctx: UserContext = { attributes, stickyBucketAssignmentDocs, saveStickyBucketAssignmentDoc: async (doc: StickyAssignmentsDocument) => {}, trackingCallback, trackedExperiments, } growthbook.evalFeature(fid, ctx); }) }, []); return null; } ``` **3. Implement tracking using <GrowthbookTrackingClient>** ex: `app/layout.tsx` ```tsx // ... import { myFlag, myExperimentFlag } from '@/flags'; import { GrowthbookTracking } from '@/components/growthbook/client-side-tracking/growthbook-tracking.tsx'; export default function Layout() { // reference the underlying ids for evaluated flags const featureIds = ["my_flag", "my_experiment_flag"]; return ( ); } ``` Follow the [example implementation here ](https://github.com/vercel/examples/blob/main/flags-sdk/growthbook/components/growthbook/client-side-tracking/growthbook-tracking.tsx). ## Examples - [Next.js 15+ example using app router ](https://github.com/vercel/examples/tree/main/flags-sdk/growthbook) ## Supported Features --- ## Node.js SDK import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Node.js We officially support Node 18 and above. ## Installation Install with a package manager ```bash npm2yarn npm install --save @growthbook/growthbook ``` ## Quick Usage First, create and instantiate a singleton GrowthBook client. You can do this as part of your server startup script. ```js const gbClient = new GrowthBookClient({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123" }); await gbClient.init({timeout: 3000}); ``` Then, you can evaluate a feature anywhere in your app by providing a feature key and a `userContext`. ```js // User context with attributes that feature flags can use for targeting const userContext = { attributes: { id: "123", } } // Boolean on/off flags if (gbClient.isOn("my-feature", userContext)) { console.log("My feature is on!"); } // String, Number, or JSON flags const value = gbClient.getFeatureValue("my-string-feature", "fallback", userContext); console.log(value); ``` ### Express Middleware In Express, it can be helpful to add a user-scoped GrowthBook instance. This will let you define a `userContext` object once instead of passing it into every feature evaluation call throughout your app. This can be accomplished easily with a middleware. ```js app.use((req, res, next) => { // Define your user context once const userContext = { attributes: { url: req.url, id: req.user.id } } // Create a scoped GrowthBook instance and store in req req.growthbook = gbClient.createScopedInstance(userContext); next(); }); ``` Now, you can use this scoped instance from any route without needing to pass in `userContext` again. ```js app.get("/", (req, res) => { // No need to pass in userContext to a scoped instance req.growthbook.isOn("my-feature"); req.growthbook.getFeatureValue("my-string-feature", "fallback"); }); ``` Scoped instances were first introduced in SDK version 1.3.1. ## Plugins GrowthBook comes with a number of built-in plugins that add additional functionality. You can enable these by passing them into the GrowthBook client's `createScopedInstance()` factory method or into the GrowthBook constructor. Plugins require GrowthBook version 1.4.0 or higher, released in Feb 2025. ### DevTools for back-end The [GrowthBook DevTools Browser Extension](/tools/chrome-extension) now supports debugging back-end environments. This includes sending overrides (attribute, feature, and experiment) from the browser extension to the back end, and also displaying back-end logged events (feature evaluations, experiment exposure events). We provide first-class devtools support via plugins and helper functions for a variety of Node.js-based environments, including: - Next.js - Express - Node.js (platform-agnostic) See the [Implementing DevTools guide](#implementing-devtools) below for more information. ### GrowthBook class (legacy) The `GrowthBookClient` class was introduced in SDK version 1.3.0 to improve performance in back-end environments by up to 3x. It does this by re-using the core instance across many different user requests. For backwards compatibility, we still support the old way of doing things with the `GrowthBook` class, where you would create a new instance for every incoming request. ```js // Example using Express app.use(function(req, res, next) { // Create a GrowthBook instance and store in the request req.growthbook = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", // Also include user/route specific settings attributes: { id: req.user.id } }); // Clean up at the end of the request res.on('close', () => req.growthbook.destroy()); // Wait for features to load (will be cached in-memory for future requests) req.growthbook.init({timeout: 1000}).then(() => next()) }); ``` The `GrowthBook` class is identical to the one used in client-side environments. Check out the [Client-side JavaScript SDK docs](/lib/js) for more info on all of the settings and methods available. ## Loading Features and Experiments In order for the GrowthBook SDK to work, it needs to have feature and experiment definitions from the GrowthBook API. There are a few ways to get this data into the SDK. ### Built-in Fetching and Caching If you pass an `apiHost` and `clientKey` into the `GrowthBookClient` constructor, it will handle the network requests, caching, retry logic, etc. for you automatically. ```ts const gbClient = new GrowthBookClient({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", }); // Wait for features to be downloaded with a timeout (in ms) await gbClient.init({ timeout: 2000 }); ``` #### Error Handling In the case of network issues that prevent the features from downloading in time, the `init` call will not throw an error. Instead, it will stay in the default state where every feature evaluates to `null`. You can still get access to the error if needed: ```ts const res = await gbClient.init({ timeout: 1000 }); if (res.error) { throw res.error; } ``` The return value of init has 3 properties: - **success** - `true` if the GrowthBook instance was populated with features/experiments. Otherwise `false` - **source** - Where this result came from. One of the following values: `network`, `cache`, `init`, `error`, or `timeout` - **error** - If success is `false`, this will contain an `Error` object with more details about the error ### Custom Integration If you prefer to handle the network and caching logic yourself, you can pass in a full JSON "payload" directly into the SDK. For example, you might store features in Postgres or Redis. ```ts await gbClient.init({ payload: { features: { "feature-1": {...}, "feature-2": {...}, "another-feature": {...}, } } }) ``` The data structure for "payload" is exactly the same as what is returned by the GrowthBook SDK endpoints and webhooks. Note: you don't need to specify `clientKey` or `apiHost` on your GrowthBook instance since no network requests are being made in this case. #### Synchronous Init There is a alternate synchronous version of init named `initSync`, which can be useful in some environments. There are some restrictions/differences: - You MUST pass in `payload` - The `payload` MUST NOT have encrypted features or experiments - The return value is the GrowthBook instance to enable easy method chaining - If using sticky bucketing, you should use an instance of `StickyBucketServiceSync`, such as `ExpressCookieStickyBucketService`. ```ts const gbClient = new GrowthBookClient().initSync({ payload: { features: {...} } }); ``` ## Refreshing Features By default, the GrowthBookClient will only fetch features once during initialization. This works great for short-running processes (e.g. serverless functions). For long-running Node.js processes, there are 2 main approaches to keeping feature defintiions up-to-date. 1. Streaming 2. Polling ### Streaming Updates The GrowthBook SDK supports streaming with Server-Sent Events (SSE). When enabled, changes to features within GrowthBook will be streamed to the SDK in realtime as they are published. This is only supported on GrowthBook Cloud or if running a GrowthBook Proxy Server. Node.js does not natively support SSE, but there is a small library you can install: ```bash npm2yarn npm install --save eventsource ``` Then, configure the polyfill and enable streaming during init: ```js const { setPolyfills } = require("@growthbook/growthbook"); // Configure GrowthBook to use the eventsource library setPolyfills({ EventSource: require("eventsource"), }); const gbClient = new GrowthBookClient({ clientKey: "sdk-abc123" }); // Enable streaming in the init call await gbClient.init({ streaming: true }); ``` This will make an initial network request to download the features payload from the GrowthBook API. Then, it will open a streaming connection to listen to updates. Every call to evaluate feature flags will use the latest available payload. ### Polling Updates If your environment doesn't support streaming updates or you want a simpler option, you can use a polling approach: ```js // Refresh once every 5 minutes setInterval(() => gbClient.refreshFeatures(), 5*60*1000); ``` We don't recommend polling more often than once per minute. If you need faster updates, use Streaming (above). ## Caching The JavaScript SDK has 2 caching layers: 1. In-memory cache (enabled by default) 2. Persistent localStorage cache (disabled by default, requires configuration) ### Configuring Local Storage Here is an example of using Redis as your persistent localStorage cache: ```js const { setPolyfills } = require("@growthbook/growthbook"); setPolyfills({ localStorage: { // Example using Redis getItem: (key) => redisClient.get(key), setItem: (key, value) => redisClient.set(key, value), } }); ``` ### Cache Settings There are a number of cache settings you can configure within GrowthBook. Below are all of the default values. You can call `configureCache` with a subset of these fields and the rest will keep their default values. ```ts import { configureCache } from "@growthbook/growthbook"; configureCache({ // The localStorage key the cache will be stored under cacheKey: "gbFeaturesCache", // Consider features stale after this much time (60 seconds default) staleTTL: 1000 * 60, // Cached features older than this will be ignored (24 hours default) maxAge: 1000 * 60 * 60 * 24, // Set to `true` to completely disable both in-memory and persistent caching disableCache: false, }) ``` ### Skip Cache The cache layers apply to both the `init` and `refreshFeatures` methods. Both of these accept a `skipCache: true` option to bypass the cache layers if desired. ## Experimentation (A/B Testing) In order to run A/B tests, you need to set up a tracking callback function. This is called every time a user is put into an experiment and can be used to track the exposure event in your analytics system (Segment, Mixpanel, GA, etc.). You can specify this globally in your GrowthBookClient class and/or per-request in your user context. We recommend using a global callback if you plan to track events directly from Node and using a user context callback if you plan to send events to the front-end to fire. Starting in version 1.7.0, tracking callbacks receive a third `userContext` argument: a `TrackingUserContext` containing the `attributes` and optional `url` used when the experiment was evaluated. We recommend recording attributes from this argument so exposure events consistently reflect the attributes GrowthBook used for targeting and assignment. ```js // Callback configured globally const gbClient = new GrowthBookClient({ trackingCallback: (experiment, result, userContext) => { const userId = userContext.attributes.id; console.log("Viewed Experiment", userId, { experimentId: experiment.key, variationId: result.key, attributes: userContext.attributes, }); } }); // Callback configured in the user context const userContext = { attributes: { id: req.user.id }, trackingCallback: (experiment, result, userContext) => { console.log("Viewed Experiment", userContext.attributes.id, { experimentId: experiment.key, variationId: result.key, attributes: userContext.attributes, }); } } ``` ### Feature Flag Experiments There is nothing special you have to do for feature flag experiments. Just evaluate the feature flag like you would normally do. If the user is put into an experiment as part of the feature flag, it will call the `trackingCallback` automatically in the background. ```js // If this has an active experiment and the user is included, // it will call trackingCallback automatically const newLogin = gbClient.isOn("new-signup-form", userContext); ``` If the experiment came from a feature rule, `result.featureId` in the trackingCallback will contain the feature id, which may be useful for tracking/logging purposes. ### Deferred Tracking Sometimes, you aren't able to track analytics events from Node.js and you need to do it from the front-end instead. In this case, define a trackingCallback in the user context, queue up events, and serialize it in your response. Here's an example: ```js // Middleware app.use((req, res, next) => { // Queue up tracking calls and store in the request req.trackingData = []; const userContext = { attributes: {id: "123"}, trackingCallback: (experiment, result, userContext) => { req.trackingData.push({experiment, result, user: userContext}); } } req.growthbook = gbClient.createScopedInstance(userContext); next(); }); // Serialize and include in your response app.get("/", (req, res) => { res.send(` (function() { const data = ${JSON.stringify(req.trackingData)}; data.forEach(({experiment, result, user}) => { // Example using Segment.io analytics.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, attributes: user.attributes, }); }) })(); ... `); }) ``` If you have also integrated the GrowthBook SDK on your front-end, there's a helper method you can use instead of manually firing tracking calls. ```ts gb.setDeferredTrackingCalls(data); gb.fireDeferredTrackingCalls(); ``` This will use the `trackingCallback` configured on your front-end GrowthBook instance. ### Sticky Bucketing Sticky bucketing ensures that users see the same experiment variant, even when user session, user login status, or experiment parameters change. See the [Sticky Bucketing docs](/app/sticky-bucketing) for more information. If your organization and experiment supports sticky bucketing, you must implement an instance of the `StickyBucketService` to use Sticky Bucketing. The JS SDK exports several implementations of this service for common use cases, or you may build your own: - `ExpressCookieStickyBucketService` — For NodeJS/Express controller-level bucket persistence using browser cookies; intended to be interoperable with `BrowserCookieStickyBucketService`. Assumes `cookie-parser` is implemented (can be polyfilled). Cookie attributes can also be configured. The default cookie expiry is 180 days; override by passing `maxAge: {ms}` into the constructor's `cookieAttributes`. - `RedisStickyBucketService` — For NodeJS Redis-based bucket persistence. Requires an `ioredis` Redis client instance to be passed in. - Build your own — Implement the abstract `StickyBucketService` class and connect to your own data store, or custom wrap multiple service implementations (ex: read/write to both cookies and Redis). Implementing most StickyBucketService implementations is straightforward and works with minimal setup. For instance, to use the `ExpressCookieStickyBucketService`: ```js const { ExpressCookieStickyBucketService } = require("@growthbook/growthbook"); app.use((req, res, next) => { const stickyBucketService = new ExpressCookieStickyBucketService({ req, res }); const userContext = await gbClient.applyStickyBuckets({ attributes: { id: req.user.id } }, stickyBucketService); req.growthbook = gbClient.createScopedInstance(userContext); next(); }); ``` ## TypeScript When used in a TypeScript project, GrowthBook includes basic type inference out of the box: ```ts // Type will be `string` based on the fallback provided ("blue") const color = gbClient.getFeatureValue("button-color", "blue", userContext); // You can manually specify types as well // feature.value will be type `number` const feature = gbClient.evalFeature("font-size", userContext); console.log(feature.value); ``` ### Strict Typing If you want to enforce stricter types in your application, you can do that when creating the GrowthBook instance: ```ts // Define all your feature flags and types here interface AppFeatures { "button-color": string; "font-size": number; "newForm": boolean; } // Pass into the GrowthBook instance const gbClient = new GrowthBookClient({ ... }); ``` Now, all feature flag methods will be strictly typed. ```ts // feature.value will be type `number` const feature = gbClient.evalFeature("font-size", userContext); console.log(feature.value); // Typos will cause compile-time errors gbClient.isOn("buton-color", userContext); // "buton" instead of "button" ``` Instead of defining the `AppFeatures` interface manually like above, you can auto-generate it from your GrowthBook account using the [GrowthBook CLI](/tools/cli). ## Updating As a general philosophy, we aim to keep the SDK 100% backwards compatible at all times. View the [Changelog](https://github.com/growthbook/growthbook/blob/main/packages/sdk-js/CHANGELOG.md) for a complete list of all SDK changes. ## GrowthBook Instance (reference) ### Attributes You can specify attributes about the current user and request. These are used for two things: 1. Feature targeting (e.g. paid users get one value, free users get another) 2. Assigning persistent variations in A/B tests (e.g. user id "123" always gets variation B) The following are some commonly used attributes, but use whatever makes sense for your application. ```ts const userContext = { attributes: { id: "123", loggedIn: true, deviceId: "abc123def456", company: "acme", paid: false, url: "/pricing", browser: "chrome", mobile: false, country: "US", }, }; ``` #### Global Attributes Sometimes there are global attributes that apply to all users. For example, the ip of your server. These can be specified on the GrowthBookClient instance and will be merged with any attributes in the user context. ```js const gbClient = new GrowthBookClient({ globalAttributes: { serverIp: "10.1.1.1" } }) ``` #### Secure Attributes When _secure attribute hashing_ is enabled, all targeting conditions in the SDK payload referencing attributes with datatype `secureString` or `secureString[]` will be anonymized via SHA-256 hashing. This allows you to safely target users based on sensitive attributes. You must enable this feature in your SDK Connection for it to take effect. If your SDK Connection has secure attribute hashing enabled, you will need to manually hash any `secureString` or `secureString[]` attributes that you pass into the GrowthBook SDK. To hash an attribute, use a cryptographic library with SHA-256 support, and compute the SHA-256 hashed value of your attribute _plus_ your organization's secure attribute salt. First, define a sha256 function using Node's built-in crypto support. ```js function sha256(str) { return crypto.createHash("sha256").update(str).digest("hex"); } ``` Then, you can use this when setting attributes. ```js const salt = "f09jq3fij"; // Your organization's secure attribute salt (see Organization Settings) // hashing a secureString attribute const userEmail = sha256(salt + user.email); // hashing an secureString[] attribute const userTags = user.tags.map(tag => sha256(salt + tag)); const userContext = { attributes: { id: user.id, loggedIn: true, email: userEmail, tags: userTags, } } ``` ### Feature Usage Callback GrowthBook can fire a callback whenever a feature is evaluated for a user. This can be useful to update 3rd party tools like NewRelic or DataDog. Like with the `trackingCallback`, this can be defined on either the GrowthBookClient instance or as part of the user context. When defined on the GrowthBookClient instance, a 3rd argument with the user context will be sent so you can identify which user evaluated a feature. ```js // Callback configured globally const gbClient = new GrowthBookClient({ onFeatureUsage: (featureKey, result, userContext) => { console.log(`${featureKey}=${result.value} for user ${userContext.attributes.id}`); } }); // Callback configured on the userContext const userContext ={ onFeatureUsage: (featureKey, result) => { console.log(`${featureKey}=${result.value}`); }, }; ``` The `result` argument is the same thing returned from `evalFeature`. Note: If you evaluate the same feature multiple times (and the value doesn't change), the callback will only be fired the first time. ### evalFeature In addition to the `isOn` and `getFeatureValue` helper methods, there is the `evalFeature` method that gives you more detailed information about why the value was assigned to the user. ```ts // Get detailed information about the feature evaluation const result = gbClient.evalFeature("my-feature", userContext); // The value of the feature (or `null` if not defined) console.log(result.value); // Why the value was assigned to the user // One of: `override`, `unknownFeature`, `defaultValue`, `force`, or `experiment` console.log(result.source); // The string id of the rule (if any) which was used console.log(result.ruleId); // Information about the experiment (if any) which was used console.log(result.experiment); // The result of the experiment (or `undefined`) console.log(result.experimentResult); ``` ### Inline Experiments Instead of declaring all features up-front in the payload and referencing them by ids in your code, you can also just run an experiment directly. This is done with the `runInlineExperiment` method: ```js // These are the only required options const { value } = gbClient.runInlineExperiment({ key: "my-experiment", variations: ["red", "blue", "green"], }, userContext); ``` #### Customizing the Traffic Split By default, this will include all traffic and do an even split between all variations. There are 2 ways to customize this behavior: ```js // Option 1: Using weights and coverage gbClient.runInlineExperiment({ key: "my-experiment", variations: ["red", "blue", "green"], // Only include 10% of traffic coverage: 0.1, // Split the included traffic 50/25/25 instead of the default 33/33/33 weights: [0.5, 0.25, 0.25], }, userContext); // Option 2: Specifying ranges gbClient.runInlineExperiment({ key: "my-experiment", variations: ["red", "blue", "green"], // Identical to the above // 5% of traffic in A, 2.5% each in B and C ranges: [ [0, 0.05], [0.5, 0.525], [0.75, 0.775], ], }, userContext); ``` #### Hashing We use deterministic hashing to assign a variation to a user. We hash together the user's id and experiment key, which produces a number between `0` and `1`. Each variation is assigned a range of numbers, and whichever one the user's hash value falls into will be assigned. You can customize this hashing behavior: ```js gbClient.runInlineExperiment({ key: "my-experiment", variations: ["A", "B"], // Which hashing algorithm to use // Version 2 is the latest and the one we recommend hashVersion: 2, // Use a different seed instead of the experiment key seed: "abcdef123456", // Use a different user attribute (default is `id`) hashAttribute: "device_id", }, userContext); ``` **Note**: For backwards compatibility, if no `hashVersion` is specified, it will fall back to using version `1`, which is deprecated. In the future, version `2` will become the default. We recommend specifying version `2` now for all new experiments to avoid migration issues down the line. #### Meta Info You can also define meta info for the experiment and/or variations. These do not affect the behavior, but they are passed through to the `trackingCallback`, so they can be used to annotate events. ```js gbClient.runInlineExperiment({ key: "results-per-page", variations: [10, 20], // Experiment meta info name: "Results per Page", phase: "full-traffic" // Variation meta info meta: [ { key: "control", name: "10 Results per Page", }, { key: "variation", name: "20 Results per Page", }, ] }, userContext) ``` #### Mutual Exclusion Sometimes you want to run multiple conflicting experiments at the same time. You can use the `filters` setting to run mutually exclusive experiments. We do this using deterministic hashing to assign users a value between 0 and 1 for each filter. ```js // Will include 60% of users - ones with a hash between 0 and 0.6 gbClient.runInlineExperiment({ key: "experiment-1", variation: [0, 1], filters: [ { seed: "pricing", attribute: "id", ranges: [[0, 0.6]] } ] }, userContext); // Will include the other 40% of users - ones with a hash between 0.6 and 1 gbClient.runInlineExperiment({ key: "experiment-2", variation: [0, 1], filters: [ { seed: "pricing", attribute: "id", ranges: [[0.6, 1.0]] } ] }, userContext); ``` #### Holdout Groups To use global holdout groups, use a nested experiment design: ```js // The value will be `true` if in the holdout group, otherwise `false` const holdout = gbClient.runInlineExperiment({ key: "holdout", variations: [true, false], // 10% of users in the holdout group weights: [0.1, 0.9] }, userContext); // Only run your main experiment if the user is NOT in the holdout if (!holdout.value) { const res = gbClient.runInlineExperiment({ key: "my-experiment", variations: ["A", "B"] }, userContext) } ``` #### Targeting Conditions You can also define targeting conditions that limit which users are included in the experiment. These conditions are evaluated against the `attributes` passed into the user context. The syntax for conditions is based on the MongoDB query syntax and is straightforward to read and write. For example, if the attributes are: ```json { "id": "123", "browser": { "vendor": "firefox", "version": 94 }, "country": "CA" } ``` The following condition would evaluate to `true` and the user would be included in the experiment: ```js gbClient.runInlineExperiment({ key: "my-experiment", variation: [0, 1], condition: { "browser.vendor": "firefox", "country": { "$in": ["US", "CA", "IN"] } } }, userContext) ``` #### Inline Experiment Return Value A call to `runInlineExperiment` returns an object with a few useful properties: ```ts const { value, key, name, variationId, inExperiment, hashUsed, hashAttribute, hashValue, } = gbClient.runInlineExperiment({ key: "my-experiment", variations: ["A", "B"], }, userContext); // If user is included in the experiment console.log(inExperiment); // true or false // The index of the assigned variation console.log(variationId); // 0 or 1 // The value of the assigned variation console.log(value); // "A" or "B" // The key and name of the assigned variation (if specified in `meta`) console.log(key); // "0" or "1" console.log(name); // "" // If the variation was randomly assigned by hashing console.log(hashUsed); // true or false // The user attribute that was hashed console.log(hashAttribute); // "id" // The value of that attribute console.log(hashValue); // e.g. "123" ``` The `inExperiment` flag will be false if the user was excluded from being part of the experiment for any reason (e.g. failed targeting conditions). The `hashUsed` flag will only be true if the user was randomly assigned a variation. If the user was forced into a specific variation instead, this flag will be false. ## Implementing DevTools The DevTools plugins are designed to work both GrowthBookClient and GrowthBook (legacy) SDK instances. Your SDK must be in dev mode (`enableDevMode: true`) for the plugin to work. ### Helper Functions The framework-specific sections below (Next.js, Express, and platform-agnostic Node.js) all rely on the same two helper functions, exported from `@growthbook/growthbook/plugins`, to send evaluation logs back to the DevTools extension. #### `getDebugScriptContents(gb, source?)` Returns a `` body that pushes the SDK's current dev-mode logs (feature evaluations, experiment exposures, decrypted payload, and attributes) onto `window._gbdebugEvents`. The GrowthBook DevTools extension reads this array from the page to display back-end evaluation results. This is the only contract between your server and the extension for back-end logging. ```ts import { getDebugScriptContents } from "@growthbook/growthbook/plugins"; const scriptBody = getDebugScriptContents(growthbook, "my-service"); // scriptBody === '(window._gbdebugEvents = (window._gbdebugEvents || [])).push({...});' ``` Notes: - Returns an **empty string** (not `null`/`undefined`) when `enableDevMode` is off, or when the instance has no evaluation logs yet. Always check for a non-empty result before injecting a `` tag, to avoid emitting an empty ``. - It snapshots `gb.logs` **at call time**. Call it _after_ you've finished evaluating every feature/experiment you want visible in DevTools for that request, not before. - `source` is a free-text label (e.g. `"nextjs"`, `"express"`, your service name) shown in the DevTools UI to identify which back-end emitted the event. It has no effect on behavior. - The returned string is raw JavaScript, not HTML. Wrap it in your own `` tags: ```ts const debugScript = getDebugScriptContents(growthbook, "my-service"); if (debugScript) { responseHtml += `${debugScript}`; } ``` - If your platform caches or shares rendered responses across requests/users (e.g. an edge/CDN worker), make sure this injection happens **after** any cache read/write step, not before. Otherwise one visitor's debug logs can end up baked into a response served to other visitors. #### `getDebugEvent(gb, source?)` The lower-level function `getDebugScriptContents` wraps. It returns the raw `LogEvent` object (`{ logs, sdkInfo }`) instead of a serialized script string. Use this directly if you want to transport the event yourself (e.g. via a response header, as shown in the Next.js middleware example below) rather than embedding it as an inline script. ### Next.js #### RSCs and SSR For RSCs (React server components) or SSR (server-side rendering), in order to apply SDK overrides from DevTools, pass the readonly request cookies and search params into the `devtoolsNextjsPlugin` plugin. In Next 15+, these may need to be awaited. ```js import { cookies } from "next/headers"; import { devtoolsNextjsPlugin } from "@growthbook/growthbook/plugins"; export default async function Home({ searchParams }) { const requestCookies = await cookies(); const awaitedSearchParams = await searchParams; const growthbook = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", enableDevMode: true, plugins: [ devtoolsNextjsPlugin({ searchParams: awaitedSearchParams, requestCookies }) ], }); return ( Hello World ); } ``` Note the component named `` which is used to hydrate SDK information and log events back into the DevTools extension. When using this component, DevTools will automatically ingest any data generated by the `getDebugScriptContents()` method. You may implement it like this: ```js import { getDebugScriptContents } from "@growthbook/growthbook/plugins"; export function DevtoolsLogger({ growthbook }) { const source = "nextjs"; return ( ); } ``` #### Middleware and API routes For middleware implementations or API routes, pass the `request` object to `devtoolsNextjsPlugin` instead of searchParams and requestCookies. A middleware example is below; it can be modified to be an API route: ```js import { NextResponse } from 'next/server'; import { devtoolsNextjsPlugin } from "@growthbook/growthbook/plugins"; export function middleware(request) { const response = NextResponse.next(); const growthbook = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", enableDevMode: true, plugins: [ devtoolsNextjsPlugin({ request }) ], }); // etc... response.headers.set( "x-gbdebug-events", JSON.stringify(getDebugEvent(req.growthbook, "nextjs_middleware")) ); return response; } ``` For passing the debug events back to DevTools, this middleware example assumes the receiving end has access to headers. This may not always be the case (e.g. redirects); you may need to rely on cookies or an encoded URL parameter. For API routes or middleware redirects, you are responsible for parsing the `x-debug-events` header value on the front-end and pushing it to `window._gbdebugEvents`. You may wish to import the `getDebugScriptContents()` method on the front-end to ingest these debug events. An example integration may look like this: ```js const response = await fetch(apiUrl); const event = response.headers.get("x-gbdebug-events"); if (event) { window._gbdebugEvents?.push(event); } ``` ### Express In an Express.js context, in order to apply SDK overrides from DevTools, pass the request object into the `devtoolsExpressPlugin` plugin. You must use the standard Express **cookie-parser** middleware (npm: `cookie-parser`) in order to use this plugin. ```js import { devtoolsExpressPlugin, getDebugEvent, } from "@growthbook/growthbook/plugins"; app.use((req, res, next) => { const userContext = { attributes: { url: req.url, id: req.user.id }, enableDevMode: true, }; // Create a scoped GrowthBook instance and store in req req.growthbook = gbClient.createScopedInstance( userContext, [ devtoolsExpressPlugin({ request: req }) ] ); res.set("x-gbdebug-events", getDebugEvent(req.growthbook, "express")); // etc... next(); }); ``` Similar to the Next.js middleware example above, you are responsible for parsing the `x-debug-events` header value on the front-end and pushing it to `window._gbdebugEvents`. You may use `getDebugScriptContents()` on the front-end to help. ### Node.js (platform agnostic) We provide a simple `devtoolsPlugin` for other Node.js back-end implementations. This plugin is a stub for the other platform-specific plugins (Next.js, Express); however, it may be used directly. When using this plugin, you are responsible for building the `devtoolsState` object by parsing the `_gbdebug` cookie and the `_gbdebug` URL query parameter. `devtoolsState` represents the overrides applied to attributes, features, and experiments; it takes the form: ```ts type DevtoolsState = { attributes?: Record; features?: Record; experiments?: Record; }; ``` Note: unlike the query param (already decoded by most URL parsers), the `_gbdebug` **cookie** value is percent-encoded. Decode it before parsing, or overrides will silently fail to apply: ```ts import { devtoolsPlugin } from "@growthbook/growthbook/plugins"; function getDevtoolsState(request): DevtoolsState { try { const raw = new URL(request.url, "http://localhost").searchParams.get("_gbdebug") ?? decodeURIComponent(getCookie(request, "_gbdebug") ?? ""); return raw ? JSON.parse(raw) : {}; } catch { return {}; } } const growthbook = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", enableDevMode: true, plugins: [devtoolsPlugin(getDevtoolsState(request))], }); ``` Similar to the Express and Next.js middleware examples above, you can generate your own `_gbdebugEvents` payload using `getDebugEvent()`. ## Examples - [Typescript example app with strict typing ](https://github.com/growthbook/examples/tree/main/vanilla-typescript). ## Supported Features --- ## OpenFeature Providers import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' # OpenFeature Providers [OpenFeature](https://openfeature.dev) is a [CNCF-incubating](https://www.cncf.io/projects/openfeature/) open standard for feature flagging. GrowthBook provides OpenFeature provider SDKs for **Python**, **Go**, **.NET**, and **Java** — listed in the [OpenFeature ecosystem](https://openfeature.dev/ecosystem/?instant_search%5Bquery%5D=growthbook). ## Prerequisites - A GrowthBook account ([cloud](https://app.growthbook.io) or [self-hosted](/self-host)) - An **SDK Connection** key — create one under **Settings → SDK Connections** in the GrowthBook UI - The API host: - Cloud: `https://cdn.growthbook.io` - Self-hosted: your own GrowthBook URL (e.g. `https://growthbook.example.com`) ## Why Use OpenFeature with GrowthBook? OpenFeature defines a vendor-neutral interface for evaluating feature flags. Using a GrowthBook OpenFeature provider means: - **Vendor portability** — Your flag evaluation code is identical regardless of which provider is active. Switching from or to GrowthBook only requires changing the provider bootstrap, not a single flag call. - **Standardized API** — Teams use the same OpenFeature interface they already know across all services. - **Ecosystem compatibility** — OpenFeature-aware tooling — including [OpenTelemetry hooks](https://openfeature.dev/docs/reference/technologies/observability/opentelemetry) and other instrumentation — works automatically with GrowthBook. - **Polyglot consistency** — The same conceptual API spans Python, Go, .NET, Java (and more), giving teams a single mental model. :::tip When to use the native GrowthBook SDK instead The native GrowthBook SDKs expose features that go beyond the OpenFeature interface: Sticky Bucketing, Visual Editor experiments, inline experiment definitions, and real-time SSE streaming. If you need those capabilities, use the [native SDK](/lib) for your language instead. ::: ## Installation & Setup **GitHub:** [growthbook-openfeature-provider-python](https://github.com/growthbook/growthbook-openfeature-provider-python)  |  **PyPI:** [growthbook-openfeature-provider](https://pypi.org/project/growthbook-openfeature-provider/) **Requirements:** Python 3.9+ ```bash pip install growthbook-openfeature-provider ``` **Async setup (recommended for async frameworks like FastAPI, aiohttp):** ```python import asyncio from openfeature.api import OpenFeatureAPI from openfeature.evaluation_context import EvaluationContext from growthbook_openfeature_provider import GrowthBookProvider, GrowthBookProviderOptions async def main(): provider = GrowthBookProvider(GrowthBookProviderOptions( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" )) await provider.initialize() OpenFeatureAPI.set_provider(provider) client = OpenFeatureAPI.get_client("my-app") # ... evaluate flags ... await provider.close() asyncio.run(main()) ``` **Synchronous setup:** ```python from openfeature.api import OpenFeatureAPI from growthbook_openfeature_provider import GrowthBookProvider, GrowthBookProviderOptions provider = GrowthBookProvider(GrowthBookProviderOptions( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" )) provider.initialize_sync() OpenFeatureAPI.set_provider(provider) client = OpenFeatureAPI.get_client("my-app") ``` **GitHub:** [growthbook-openfeature-provider-go](https://github.com/growthbook/growthbook-openfeature-provider-go)  |  **pkg.go.dev:** [growthbook-openfeature-provider-go](https://pkg.go.dev/github.com/growthbook/growthbook-openfeature-provider-go) **Requirements:** Go 1.21+ ```bash go get github.com/growthbook/growthbook-openfeature-provider-go ``` ```go import ( "context" "log" gb "github.com/growthbook/growthbook-golang" gbprovider "github.com/growthbook/growthbook-openfeature-provider-go" "github.com/open-feature/go-sdk/openfeature" ) // Create the underlying GrowthBook client gbClient, err := gb.NewClient(context.Background(), gb.WithAPIHost("https://cdn.growthbook.io"), gb.WithClientKey("sdk-abc123"), ) if err != nil { log.Fatal("GrowthBook client initialization failed: ", err) } defer gbClient.Close() // Wrap it in the OpenFeature provider and register it provider := gbprovider.NewProvider(gbClient) if err = openfeature.SetProvider(provider); err != nil { log.Fatal("Failed to set OpenFeature provider: ", err) } client := openfeature.NewClient("my-app") ``` **GitHub:** [growthbook-openfeature-provider-dot-net](https://github.com/growthbook/growthbook-openfeature-provider-dot-net)  |  **NuGet:** [GrowthBook.OpenFeature](https://www.nuget.org/packages/GrowthBook.OpenFeature) **Requirements:** .NET 8+ or .NET Framework 4.6.2+ ```bash dotnet add package GrowthBook.OpenFeature ``` ```csharp using GrowthBook.OpenFeature; using OpenFeature; using OpenFeature.Model; var provider = new GrowthBookProvider( clientKey: "sdk-abc123", apiHostUrl: "https://cdn.growthbook.io" ); Api.Instance.SetProvider(provider); var client = Api.Instance.GetClient(); ``` **GitHub:** [growthbook-openfeature-provider-java](https://github.com/growthbook/growthbook-openfeature-provider-java)  |  **Maven Central:** [growthbook-openfeature-provider-java](https://central.sonatype.com/artifact/com.github.growthbook/growthbook-openfeature-provider-java) **Requirements:** Java 8+ ```xml com.github.growthbook growthbook-openfeature-provider-java 0.0.1 ``` ```gradle implementation group: 'com.github.growthbook', name: 'growthbook-openfeature-provider-java', version: '0.0.1' ``` ```java import io.github.growthbook.GrowthBookProvider; import io.github.growthbook.Options; import dev.openfeature.sdk.OpenFeatureAPI; import dev.openfeature.sdk.Client; Options options = Options.builder() .apiHost("https://cdn.growthbook.io") .clientKey("sdk-abc123") .build(); GrowthBookProvider provider = new GrowthBookProvider(options); OpenFeatureAPI.getInstance().setProvider(provider); Client client = OpenFeatureAPI.getInstance().getClient(); ``` ## Evaluation Context (Targeting Attributes) Pass user attributes to the OpenFeature client via an `EvaluationContext`. These become GrowthBook targeting attributes used for percentage rollouts, feature targeting rules, and experiment bucketing. The `targetingKey` maps to the user ID in GrowthBook. Additional attributes (e.g. `country`, `plan`, `email`) are matched against your feature flag targeting conditions. ```python from openfeature.evaluation_context import EvaluationContext context = EvaluationContext( targeting_key="user-123", attributes={ "country": "US", "plan": "premium", "email": "user@example.com", } ) ``` ```go evalCtx := openfeature.NewEvaluationContext("user-123", map[string]interface{}{ "country": "US", "plan": "premium", "email": "user@example.com", }) ``` ```csharp var context = new EvaluationContext( targetingKey: "user-123", new Dictionary { { "country", new Value("US") }, { "plan", new Value("premium") }, { "email", new Value("user@example.com") }, } ); ``` ```java import dev.openfeature.sdk.EvaluationContext; import dev.openfeature.sdk.MutableContext; MutableContext context = new MutableContext("user-123"); context.add("country", "US"); context.add("plan", "premium"); context.add("email", "user@example.com"); ``` ## Evaluating Feature Flags OpenFeature defines five value types. GrowthBook's feature flags map to all of them. ### Boolean The most common type — use for on/off feature gates. ```python # Synchronous enabled = client.get_boolean_value("dark-mode", False, context) # Async (returns FlagEvaluationDetails with value, reason, variant, etc.) details = await provider.resolve_boolean_details_async("dark-mode", False, context) enabled = details.value ``` ```go // Returns the value with a default fallback enabled, err := client.BooleanValue(context.Background(), "dark-mode", false, evalCtx) // Returns full evaluation details (value, reason, variant, error) details, err := client.BooleanValueDetails(context.Background(), "dark-mode", false, evalCtx) enabled := details.Value ``` ```csharp bool enabled = await client.GetBooleanValue("dark-mode", false, context); ``` ```java boolean enabled = client.getBooleanValue("dark-mode", false, context); ``` ### String Use for multi-variant flags where the value is a string (e.g. button color, layout variant). ```python variant = client.get_string_value("button-color", "blue", context) ``` ```go variant, err := client.StringValue(context.Background(), "button-color", "blue", evalCtx) ``` ```csharp string variant = await client.GetStringValue("button-color", "blue", context); ``` ```java String variant = client.getStringValue("button-color", "blue", context); ``` ### Integer & Float/Double Use for numeric feature values such as rate limits, timeouts, or pricing. ```python request_limit = client.get_integer_value("api-request-limit", 100, context) price = client.get_float_value("subscription-price", 9.99, context) ``` ```go requestLimit, err := client.IntValue(context.Background(), "api-request-limit", 100, evalCtx) price, err := client.FloatValue(context.Background(), "subscription-price", 9.99, evalCtx) ``` ```csharp int requestLimit = await client.GetIntegerValue("api-request-limit", 100, context); double price = await client.GetDoubleValue("subscription-price", 9.99, context); ``` ```java int requestLimit = client.getIntegerValue("api-request-limit", 100, context); double price = client.getDoubleValue("subscription-price", 9.99, context); ``` ### Object Use for structured feature values (e.g. configuration payloads, theme objects). ```python config = client.get_object_value("theme-config", {"color": "blue"}, context) ``` ```go // Object values are returned as map[string]interface{} config, err := client.ObjectValue(context.Background(), "theme-config", map[string]interface{}{}, evalCtx) ``` ```csharp Value config = await client.GetObjectValue("theme-config", new Value(new Dictionary()), context); ``` ```java Value config = client.getObjectValue("theme-config", new MutableStructure(), context); ``` ## Configuration Options | Option | Python | Go | .NET | Java | Description | | -------------------------- | :----: | :-: | :--: | :--: | ------------------------------------------------- | | `api_host` / `apiHost` | ✓ | ✓ | ✓ | ✓ | GrowthBook CDN or self-hosted URL | | `client_key` / `clientKey` | ✓ | ✓ | ✓ | ✓ | SDK Connection key from GrowthBook | | `decryption_key` | ✓ | — | — | — | Key for encrypted SDK endpoint payloads | | `cache_ttl` | ✓ | ✓ | — | — | Seconds before re-fetching features (default: 60) | | `enabled` | ✓ | — | ✓ | — | Enable/disable the provider entirely | | `qa_mode` | ✓ | — | — | — | Forces all experiments into the control variation | ## Shutdown & Cleanup Always shut down the provider when your application exits to close background connections. ```python await provider.close() # async # or provider.close_sync() # sync ``` ```go defer gbClient.Close() ``` ```csharp provider.Dispose(); ``` ```java provider.shutdown(); ``` ## Further Reading - [OpenFeature specification](https://openfeature.dev/specification) — the full standard - [GrowthBook on the OpenFeature ecosystem](https://openfeature.dev/ecosystem/?instant_search%5Bquery%5D=growthbook) - [Native GrowthBook SDKs](/lib) — for Sticky Bucketing, Visual Editor, SSE streaming, and more --- ## PHP SDK import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # PHP The GrowthBook PHP SDK requires PHP version 7.1 or greater. ## Installation GrowthBook is available on Composer: ```bash composer require growthbook/growthbook ``` ## Quick Usage ```php // Create a GrowthBook instance $growthbook = Growthbook\Growthbook::create() ->withAttributes([ // Targeting attributes 'id' => $userId, 'someCustomAttribute' => true ]); // Load feature flags from the GrowthBook API // Make sure to use caching in production! (see 'Loading Features' below) // Note: Prior to version 1.7.0, this method was called `loadFeatures` instead of `initialize` $growthbook->initialize("sdk-abc123", "https://cdn.growthbook.io"); // Feature gating if ($growthbook->isOn("my-feature")) { echo "It's on!"; } else { echo "It's off :("; } // Remote configuration with fallback $color = $growthbook->getValue("button-color", "blue"); echo "Click Me!"; ``` Some of the feature flags you evaluate might be running an A/B test behind the scenes which you'll want to track in your analytics system. At the end of the request, you can loop through all experiments and track them however you want to: ```php $impressions = $growthbook->getViewedExperiments(); foreach($impressions as $impression) { // Whatever you use for event tracking Segment::track([ "userId" => $userId, "event" => "Experiment Viewed", "properties" => [ "experimentId" => $impression->experiment->key, "variationId" => $impression->result->key ] ]); } ``` ### Loading Features There are 2 ways to load features into the SDK. You can use `initialize` with a Client Key and API Host. Or, you can manually fetch and cache feature flags and pass them in with the `withFeatures` method. #### initialize method The `initialize` method can fetch features from the GrowthBook API for you. By default, there is no caching enabled. You can enable it by passing any PSR16-compatible instance into the `withCache` method. **Caching is required for production usage** ```php // Any psr-16 library will work use Cache\Adapter\Apcu\ApcuCachePool; $cache = new ApcuCachePool(); $growthbook = Growthbook\Growthbook::create() ->withCache($cache); // You can optionally pass in a TTL (default 60s) $growthbook = Growthbook\Growthbook::create() ->withCache($cache, 120); // Cache for 120s instead ``` To load features, we require a PSR-17 (HttpClient) and PSR-18 (RequestFactoryInterface) compatible library like Guzzle to be installed. We will auto-discover most HTTP libraries without any configuration required, but if you prefer to specify it explicitly, you can use the `withHttpClient` method. Note - you'll need to specify both an `HttpClient` and a `RequestFactoryInterface` implementation. The `initialize` method takes 3 arguments: - `$clientKey` (required) - Get this from your SDK Connection in GrowthBook. - `$apiHost` (optional) - Defaults to `https://cdn.growthbook.io`. If self-hosting GrowthBook, set this to your API host. - `$decryptionKey` (optional) - Only required if you've enabled encryption for your SDK Connection. #### withFeatures method If you prefer to have full control over the fetching/caching behavior, you can use the `withFeatures` method instead to pass an associative array of features into the SDK. #### withSavedGroups method If you're using saved groups and not using the `initialize` method, you'll need to call `withSavedGroups` to provide the saved groups data to the SDK. ```php // From the GrowthBook API, a custom cache layer, or somewhere else $featuresJSON = '{"my-feature":{"defaultValue":true}}'; // Decode into an associative array $features = json_decode($featuresJSON, true); // Pass into the Growthbook instance $growthbook = Growthbook\Growthbook::create() ->withFeatures($features); // Optionally, if using saved groups $savedGroupsJSON = '{"myGroup": [1, 2, 3]}'; $savedGroups = json_decode($savedGroupsJSON, true); $growthbook = $growthbook->withSavedGroups($savedGroups); ``` ## The Growthbook Class The `Growthbook` class has a number of properties. These can be set using a Fluent interface or can be passed into a constructor using an associative array. Every property also has a getter method if needed. Here's an example: ```php // Using the fluent interface $growthbook = Growthbook\Growthbook::create() ->withFeatures($features) ->withAttributes($attributes); // Using the constructor $growthbook = new Growthbook\Growthbook([ 'features' => $features, 'attributes' => $attributes ]); // Getter methods print_r($growthbook->getFeatures()); print_r($growthbook->getAttributes()); ``` Note: you can also use the fluent methods (e.g. `withFeatures`) at any point to update properties. ### Attributes You can specify attributes about the current user and request. These are used for two things: 1. Feature targeting (e.g. paid users get one value, free users get another) 2. Assigning persistent variations in A/B tests (e.g. user id "123" always gets variation B) Attributes can be any JSON data type - boolean, integer, float, string, or array. ```php $attributes = [ 'id' => "123", 'loggedIn' => true, 'deviceId' => "abc123def456", 'age' => 21, 'tags' => ["tag1", "tag2"], 'account' => [ 'age' => 90 ] ]; ``` If you want to update attributes later, please note that the `withAttributes` method completely overwrites the attributes object. You can use `array_merge` if you only want to update a subset of fields: ```php // Only update the url attribute $growthbook->withAttributes(array_merge( $growthbook->getAttributes(), [ 'url' => '/checkout' ] )); ``` ### Tracking Experiments Any time an experiment is run to determine the value of a feature, you want to track that event in your analytics system. You can either do this via a callback function: ```php $trackingCallback = function ( Growthbook\InlineExperiment $experiment, Growthbook\ExperimentResult $result ) { // Segment.io example Segment::track([ "userId" => $userId, "event" => "Experiment Viewed", "properties" => [ "experimentId" => $experiment->key, "variationId" => $result->key ] ]); }; // Fluent interface $growthbook = Growthbook\Growthbook::create() ->withTrackingCallback($callback); // Using the constructor $growthbook = new Growthbook([ 'trackingCallback' => $trackingCallback ]); // Getter method $trackingCallback = $growthbook->getTrackingCallback(); ``` Or track all events at the end of the request by looping through an array: ```php $impressions = $growthbook->getViewedExperiments(); foreach($impressions as $impression) { // Segment.io example Segment::track([ "userId" => $userId, "event" => "Experiment Viewed", "properties" => [ "experimentId" => $impression->experiment->key, "variationId" => $impression->result->key ] ]); } ``` Or, you can pass the impressions onto your front-end and fire analytics events from there. To do this, simply add a block to your template (shown here in plain PHP, but similar idea for Twig, Blade, etc.). ```php getViewedExperiments() as $impression): ?> // tracking code goes here ``` Below are examples for a few popular front-end tracking libraries: #### Google Analytics ```php ga('send', 'event', 'experiment', "experiment->key ?>", "result->variationId ?>", { // Custom dimension for easier analysis 'dimension1': "experiment->key.':'.$impression->result->key ?>" } ); ``` #### Segment ```php analytics.track("Experiment Viewed", $impression->experiment->key, "variationId" => $impression->result->key ])?>); ``` #### Mixpanel ```php mixpanel.track("Experiment Viewed", $impression->experiment->key, 'Variant name' => $impression->result->key ])?>); ``` ### Logging GrowthBook can output log messages to help you debug your feature flags and experiments. We support any PSR-3 compatible logger. We implement a fluent interface (`withLogger`) as well as the standard LoggerAware interface (`setLogger`). ```php // Fluent interface $growthbook ->withLogger($logger) ->with...; // Setter $growthbook->setLogger($logger); ``` ## Using Features There are 3 main methods for interacting with features. - `$growthbook->isOn("feature-key")` returns true if the feature is on - `$growthbook->isOff("feature-key")` returns false if the feature is on - `$growthbook->getValue("feature-key", "default")` returns the value of the feature with a fallback In addition, you can use `$growthbook->getFeature("feature-key")` to get back a `FeatureResult` object with the following properties: - **value** - The JSON-decoded value of the feature (or `null` if not defined) - **on** and **off** - The JSON-decoded value cast to booleans - **source** - Why the value was assigned to the user. One of `unknownFeature`, `defaultValue`, `force`, or `experiment` - **experiment** - Information about the experiment (if any) which was used to assign the value to the user - **experimentResult** - The result of the experiment (if any) which was used to assign the value to the user ## Sticky Bucketing By default GrowthBook does not persist assigned experiment variations for a user. We rely on deterministic hashing to ensure that the same user attributes always map to the same experiment variation. However, there are cases where this isn't good enough. For example, if you change targeting conditions in the middle of an experiment, users may stop being shown a variation even if they were previously bucketed into it. Sticky Bucketing is a solution to these issues. You can provide a Sticky Bucket Service to the GrowthBook instance to persist previously seen variations and ensure that the user experience remains consistent for your users. A sample `InMemoryStickyBucketService` implementation is provided for reference, but in production you will definitely want to implement your own version using a database, cookies, or similar for persistence. Sticky Bucket documents contain three fields - `attributeName` - The name of the attribute used to identify the user (e.g. `id`, `cookie_id`, etc.) - `attributeValue` - The value of the attribute (e.g. `123`) - `assignments` - A dictionary of persisted experiment assignments. For example: `{"exp1__0":"control"}` The attributeName/attributeValue combo is the primary key. Here's an example implementation using a theoretical `db` object: ```php class InMemoryStickyBucketService extends StickyBucketService { /** @var array */ public array $docs = []; /** * @param string $attributeName * @param mixed $attributeValue * @return StickyAssignmentDocument|null */ public function getAssignments(string $attributeName, $attributeValue): ?StickyAssignmentDocument { return $this->docs[$this->getKey($attributeName, $attributeValue)] ?? null; } /** * @param StickyAssignmentDocument $doc * @return void */ public function saveAssignments(StickyAssignmentDocument $doc): void { $this->docs[$this->getKey($doc->getAttributeName(), $doc->getAttributeValue())] = $doc; } /** * @return void */ public function destroy(): void { $this->docs = []; } } // Fluent interface $growthbook = Growthbook\Growthbook::create() ->withStickyBucketing(new InMemoryStickyBucketService()); // Using the constructor $growthbook = new Growthbook([ 'stickyBucketService' => new InMemoryStickyBucketService() ]); ``` ## Inline Experiments Instead of declaring all features up-front and referencing them by ids in your code, you can also just run an experiment directly. This is done with the `$growthbook->runInlineExperiment` method: ```php $exp = Growthbook\InlineExperiment::create( "my-experiment", ["red", "blue", "green"] ); // Either "red", "blue", or "green" echo $growthbook->runInlineExperiment($exp)->value; ``` As you can see, there are 2 required parameters for experiments, a string key, and an array of variations. Variations can be any data type, not just strings. There are a number of additional settings to control the experiment behavior. The methods are all chainable. Here's an example that shows all of the possible settings: ```php $exp = Growthbook\InlineExperiment::create("my-experiment", ["red","blue"]) // Run a 40/60 experiment instead of the default even split (50/50) ->withWeights([0.4, 0.6]) // Only include 20% of users in the experiment ->withCoverage(0.2) // Targeting conditions using a MongoDB-like syntax ->withCondition([ 'country' => 'US', 'browser' => [ '$in' => ['chrome', 'firefox'] ] ]) // Use an alternate attribute for assigning variations (default is 'id') ->withHashAttribute("sessionId") // Namespaces are used to run mutually exclusive experiments // Another experiment in the "pricing" namespace with a non-overlapping range // will be mutually exclusive (e.g. [0.5, 1]) ->withNamespace("pricing", 0, 0.5); ``` ### Inline Experiment Return Value A call to `runInlineExperiment` returns an `ExperimentResult` object with a few useful properties: ```php $result = $growthbook->runInlineExperiment($exp); // If user is part of the experiment echo($result->inExperiment); // true or false // The index of the assigned variation echo($result->variationId); // e.g. 0 or 1 // The key used to identify this variation when tracking the event echo($result->key); // e.g. "control" // The value of the assigned variation echo($result->value); // e.g. "A" or "B" // If the variations was randomly assigned based on a hash echo($result->hashUsed); // true or false // The user attribute that was hashed echo($result->hashAttribute); // "id" // The value of that attribute echo($result->hashValue); // e.g. "123" ``` The `inExperiment` flag will be false if the user was excluded from being part of the experiment for any reason (e.g. failed targeting conditions). The `hashUsed` flag will only be true if the user was randomly assigned a variation. If the user was forced into a specific variation instead, this flag will be false. ## Supported Features --- ## Python SDK import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' # Python _Requires **Python 3.6** or above_ ## Installation ```bash pip install growthbook ``` ## Quick Usage ```python from growthbook import GrowthBook # User attributes for targeting and experimentation attributes = { "id": "123", "customUserAttribute": "foo" } def on_experiment_viewed(experiment, result): # Use whatever event tracking system you want analytics.track(attributes["id"], "Experiment Viewed", { 'experimentId': experiment.key, 'variationId': result.key }) # Create a GrowthBook instance gb = GrowthBook( attributes = attributes, on_experiment_viewed = on_experiment_viewed, api_host = "https://cdn.growthbook.io", client_key = "sdk-abc123" ) # Load features from the GrowthBook API with caching gb.load_features() # Simple on/off feature gating if gb.is_on("my-feature"): print("My feature is on!") # Get the value of a feature with a fallback color = gb.get_feature_value("button-color-feature", "blue") ``` ## Async Client (Recommended) **Available starting in version 1.2.0** For improved performance and better resource utilization, especially in async web applications, use the `GrowthBookClient` class. This approach provides up to **3x better performance** by reusing a single client instance across multiple requests instead of creating new instances per request. ### Basic Async Usage ```python from growthbook import GrowthBookClient, Options, UserContext, FeatureRefreshStrategy import asyncio async def main(): # Create client options options = Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # Optional: Enable real-time feature updates refresh_strategy=FeatureRefreshStrategy.SERVER_SENT_EVENTS ) # Create and initialize client client = GrowthBookClient(options) try: # Initialize the client before using it success = await client.initialize() if not success: print("Failed to initialize GrowthBook client") return # Create user context for targeting user = UserContext( attributes={ "id": "123", "country": "US", "premium": True } ) # Simple feature evaluation if await client.is_on("new-homepage", user): print("New homepage is enabled!") # Get feature value with fallback color = await client.get_feature_value("button-color", "blue", user) print(f"Button color is {color}") # Run an experiment from growthbook import Experiment result = await client.run( Experiment( key="my-test", variations=["A", "B"] ), user ) print(f"User got variation: {result.value}") finally: # Always close the client when done await client.close() # Run the async code asyncio.run(main()) ``` For web framework integration examples, see [Integration Examples](#integration-examples) below. ### Real-time Feature Updates The async client supports real-time feature updates using Server-Sent Events: ```python from growthbook import GrowthBookClient, Options, FeatureRefreshStrategy client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # Enable SSE for real-time updates refresh_strategy=FeatureRefreshStrategy.SERVER_SENT_EVENTS ) ) ``` ### Concurrency and Thread Safety The async client is designed to be thread-safe and handle concurrent requests efficiently. You can safely use a single client instance across multiple coroutines: ```python from fastapi import FastAPI from growthbook import GrowthBookClient, Options, UserContext import asyncio app = FastAPI() # Single client instance shared across all requests gb_client = GrowthBookClient(Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" )) @app.on_event("startup") async def startup(): await gb_client.initialize() @app.on_event("shutdown") async def shutdown(): await gb_client.close() @app.get("/batch") async def batch_process(user_ids: list[str]): # Safely process multiple users concurrently tasks = [] for user_id in user_ids: user = UserContext(attributes={"id": user_id}) tasks.append(gb_client.eval_feature("new-feature", user)) results = await asyncio.gather(*tasks) return {"results": results} ``` **Note**: While the client is thread-safe, you should not share a single `UserContext` instance across different requests. Create a new `UserContext` for each request to maintain proper isolation. ### Performance Benefits The `GrowthBookClient` provides significant performance improvements over the traditional per-request `GrowthBook` approach: - **3x faster** feature evaluations due to instance reuse - **Lower memory usage** by sharing feature data across requests - **Built-in caching** with configurable refresh strategies - **Real-time updates** without polling overhead - **Async/await support** for non-blocking operations ## Loading Features There are two ways to load feature flags into the GrowthBook SDK. You can either use the built-in fetching/caching logic or implement your own custom solution. ### Built-in Fetching and Caching Both the async client and traditional client support built-in fetching and caching of feature flags. For the async client, use `GrowthBookClient` with `Options`: ```python import asyncio from growthbook import GrowthBookClient, Options, FeatureRefreshStrategy async def main(): # Create client with built-in fetching and caching client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # Optional: Configure caching cache_ttl=300, # Cache for 5 minutes # Optional: Enable real-time updates refresh_strategy=FeatureRefreshStrategy.SERVER_SENT_EVENTS, # Optional: Encryption support decryption_key="your-decryption-key" ) ) try: # Initialize features (async) await client.initialize() # Features are now cached and ready to use user = UserContext(attributes={"id": "user_123"}) feature_enabled = await client.is_on("my-feature", user) print(f"Feature enabled: {feature_enabled}") finally: await client.close() asyncio.run(main()) ``` For the traditional client, pass `api_host` and `client_key` to the constructor: ```python from growthbook import GrowthBook # Create client with built-in fetching and caching gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # How long to cache features in seconds (Optional, default 60s) cache_ttl=60, # Optional: Encryption support decryption_key="your-decryption-key", # User attributes attributes={"id": "user_123"} ) # Load features (synchronous) gb.load_features() # Features are now cached and ready to use feature_enabled = gb.is_on("my-feature") print(f"Feature enabled: {feature_enabled}") # Clean up gb.destroy() ``` #### Custom Caching GrowthBook comes with a custom in-memory cache. For the traditional client, configure a custom cache globally: ```python from redis import Redis import json from growthbook import GrowthBook, AbstractFeatureCache, feature_repo class RedisFeatureCache(AbstractFeatureCache): def __init__(self): self.r = Redis(host='localhost', port=6379) self.prefix = "gb:" def get(self, key: str): data = self.r.get(self.prefix + key) # Data stored as a JSON string, parse into dict before returning return None if data is None else json.loads(data) def set(self, key: str, value: dict, ttl: int) -> None: self.r.set(self.prefix + key, json.dumps(value)) self.r.expire(self.prefix + key, ttl) # Configure GrowthBook to use your custom cache class feature_repo.set_cache(RedisFeatureCache()) # Now all GrowthBook instances will use Redis caching gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" ) gb.load_features() # Will use Redis cache ``` ### Custom Implementation If you prefer to handle the entire fetching/caching logic yourself, you can just pass in a `dict` of features from the GrowthBook API directly into the constructor: ```python # From the GrowthBook API features = {'my-feature':{'defaultValue':False}} gb = GrowthBook( features = features ) ``` Note: When doing this, you do not need to specify your `api_host` or `client_key` and you don't need to call `gb.load_features()`. ## GrowthBook class The GrowthBook constructor has the following parameters: - **enabled** (`bool`) - Flag to globally disable all experiments. Default true. - **attributes** (`dict`) - Dictionary of user attributes that are used for targeting and to assign variations - **url** (`str`) - The URL of the current request (if applicable) - **qa_mode** (`boolean`) - If true, random assignment is disabled and only explicitly forced variations are used. - **on_experiment_viewed** (`callable`) - A function that takes `experiment` and `result` as arguments. - **api_host** (`str`) - The GrowthBook API host to fetch feature flags from. Defaults to `https://cdn.growthbook.io` - **client_key** (`str`) - The client key that will be passed to the API Host to fetch feature flags - **decryption_key** (`str`) - If the GrowthBook API endpoint has encryption enabled, specify the decryption key here - **cache_ttl** (`int`) - How long to cache features in-memory from the GrowthBook API (seconds, default `60`) - **features** (`dict`) - Feature definitions from the GrowthBook API (only required if `client_key` is not specified) - **forced_variations** (`dict`) - Dictionary of forced experiment variations (used for QA) There are also getter and setter methods for features and attributes if you need to update them later in the request: ```python gb.set_features(gb.get_features()) gb.set_attributes(gb.get_attributes()) ``` ### Attributes You can specify attributes about the current user and request. These are used for two things: 1. Feature targeting (e.g. paid users get one value, free users get another) 2. Assigning persistent variations in A/B tests (e.g. user id "123" always gets variation B) Attributes can be any JSON data type - boolean, integer, float, string, list, or dict. For the async client, attributes are passed via `UserContext` for each evaluation: ```python import asyncio from growthbook import GrowthBookClient, Options, UserContext async def main(): client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" ) ) try: await client.initialize() # Create user context with attributes user = UserContext( attributes={ 'id': "123", 'loggedIn': True, 'age': 21.5, 'tags': ["tag1", "tag2"], 'account': { 'age': 90 }, 'country': 'US', 'premium': True } ) # Use features with user context feature_enabled = await client.is_on("premium-feature", user) discount = await client.get_feature_value("discount-percent", 0, user) # Different user context for another request guest_user = UserContext( attributes={ 'id': "guest_456", 'loggedIn': False, 'country': 'CA' } ) guest_feature = await client.is_on("premium-feature", guest_user) finally: await client.close() asyncio.run(main()) ``` For the traditional client, attributes are set globally on the instance: ```python from growthbook import GrowthBook # Define user attributes attributes = { 'id': "123", 'loggedIn': True, 'age': 21.5, 'tags': ["tag1", "tag2"], 'account': { 'age': 90 }, 'country': 'US', 'premium': True } # Pass into constructor gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes=attributes ) gb.load_features() # Use features (uses global attributes) feature_enabled = gb.is_on("premium-feature") discount = gb.get_feature_value("discount-percent", 0) # Or update attributes later new_attributes = { 'id': "456", 'loggedIn': False, 'country': 'CA' } gb.set_attributes(new_attributes) # Features will now use updated attributes updated_feature = gb.is_on("premium-feature") gb.destroy() ``` #### Secure Attributes When _secure attribute hashing_ is enabled, all targeting conditions in the SDK payload referencing attributes with datatype `secureString` or `secureString[]` will be anonymized via SHA-256 hashing. This allows you to safely target users based on sensitive attributes. You must enable this feature in your SDK Connection for it to take effect. If your SDK Connection has secure attribute hashing enabled, you will need to manually hash any `secureString` or `secureString[]` attributes that you pass into the GrowthBook SDK. To hash an attribute, use the `hashlib` library with SHA-256 support, and compute the SHA-256 hashed value of your attribute _plus_ your organization's secure attribute salt. For the async client, hash secure attributes before creating the `UserContext`: ```python import asyncio import hashlib from growthbook import GrowthBookClient, Options, UserContext def hash_secure_attribute(value: str, salt: str) -> str: """Helper function to hash secure attributes""" return hashlib.sha256(f"{salt}{value}".encode()).hexdigest() async def main(): # Your secure attribute salt (set in Organization Settings) salt = "f09jq3fij" client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" ) ) try: await client.initialize() # Hash secure attributes user_email = "user@example.com" user_tags = ["premium", "beta"] hashed_email = hash_secure_attribute(user_email, salt) hashed_tags = [hash_secure_attribute(tag, salt) for tag in user_tags] # Create user context with hashed secure attributes user = UserContext( attributes={ 'id': "123", 'loggedIn': True, 'email': hashed_email, # secureString 'tags': hashed_tags, # secureString[] 'country': 'US' # regular attribute } ) # Use features with secure attributes premium_feature = await client.is_on("premium-feature", user) personalized_content = await client.get_feature_value("personalized-content", {}, user) finally: await client.close() asyncio.run(main()) ``` For the traditional client, hash secure attributes before passing to the constructor: ```python import hashlib from growthbook import GrowthBook def hash_secure_attribute(value: str, salt: str) -> str: """Helper function to hash secure attributes""" return hashlib.sha256(f"{salt}{value}".encode()).hexdigest() # Your secure attribute salt (set in Organization Settings) salt = "f09jq3fij" # Hash secure attributes user_email = "user@example.com" user_tags = ["premium", "beta"] hashed_email = hash_secure_attribute(user_email, salt) hashed_tags = [hash_secure_attribute(tag, salt) for tag in user_tags] # Create GrowthBook instance with hashed secure attributes gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={ 'id': "123", 'loggedIn': True, 'email': hashed_email, # secureString 'tags': hashed_tags, # secureString[] 'country': 'US' # regular attribute } ) gb.load_features() # Use features with secure attributes premium_feature = gb.is_on("premium-feature") personalized_content = gb.get_feature_value("personalized-content", {}) gb.destroy() ``` ### Tracking Experiments Any time an experiment is run to determine the value of a feature, you want to track that event in your analytics system. For the async client, you can set up experiment tracking through the `Options`: ```python import asyncio from growthbook import GrowthBookClient, Options, UserContext, Experiment, Result def on_experiment_viewed(experiment: Experiment, result: Result, user_context: UserContext): """Synchronous callback for experiment tracking (Fire-and-Forget)""" # Use whatever event tracking system you want user_id = result.hash_attribute or "anonymous" # For async analytics, you must schedule the task on the loop # without awaiting it (Fire-and-Forget) loop = asyncio.get_running_loop() loop.create_task(track_async(user_id, experiment, result)) async def track_async(user_id, experiment, result): # This runs in the background await analytics.track_async(user_id, "Experiment Viewed", { 'experimentId': experiment.key, 'variationId': result.key, 'variationValue': result.value, 'inExperiment': result.in_experiment, 'hashUsed': result.hash_used }) print(f"Tracked experiment: {experiment.key} -> {result.value}") async def main(): # Create client with experiment tracking client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # Callback must be synchronous on_experiment_viewed=on_experiment_viewed ) ) try: await client.initialize() # Create user context user = UserContext(attributes={"id": "user_123", "premium": True}) # Feature evaluations will automatically trigger tracking feature_value = await client.get_feature_value("button-color", "blue", user) # Run inline experiments (also triggers tracking) experiment = Experiment( key="pricing-test", variations=["$9.99", "$14.99", "$19.99"] ) result = await client.run(experiment, user) print(f"Experiment result: {result.value}") finally: await client.close() asyncio.run(main()) ``` You can also use synchronous callbacks with the async client: ```python def sync_experiment_tracker(experiment: Experiment, result: Result): """Synchronous callback that works with async client""" # Send to synchronous analytics service analytics.track("Experiment Viewed", { 'experimentId': experiment.key, 'variationId': result.key, 'userId': result.hash_attribute }) client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", on_experiment_viewed=sync_experiment_tracker # Sync callback ) ) ``` For the traditional client, pass the callback to the constructor: ```python from growthbook import GrowthBook, Experiment, Result def on_experiment_viewed(experiment: Experiment, result: Result): """Callback for experiment tracking""" # Use whatever event tracking system you want user_id = result.hash_attribute or "anonymous" analytics.track(user_id, "Experiment Viewed", { 'experimentId': experiment.key, 'variationId': result.key, 'variationValue': result.value, 'inExperiment': result.in_experiment, 'hashUsed': result.hash_used }) print(f"Tracked experiment: {experiment.key} -> {result.value}") # Create GrowthBook instance with experiment tracking gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={"id": "user_123", "premium": True}, on_experiment_viewed=on_experiment_viewed ) gb.load_features() # Feature evaluations will automatically trigger tracking feature_value = gb.get_feature_value("button-color", "blue") # Run inline experiments (also triggers tracking) experiment = Experiment( key="pricing-test", variations=["$9.99", "$14.99", "$19.99"] ) result = gb.run(experiment) print(f"Experiment result: {result.value}") gb.destroy() ``` ### Tracking Plugins **Available starting in version 1.3.0** The Python SDK supports tracking plugins that provide automated event tracking with batching, error handling, and retry logic. This is the recommended approach for production applications as it handles edge cases and provides better reliability than custom tracking callbacks. #### Built-in Tracking Plugin The SDK includes a built-in tracking plugin that automatically batches and sends events: ```python import asyncio from growthbook import GrowthBookClient, Options, UserContext, GrowthBookTrackingPlugin # Custom synchronous tracking callback def my_custom_tracker(experiment, result, user_context): """Process tracking event (Fire-and-Forget)""" loop = asyncio.get_running_loop() loop.create_task(process_async(experiment, result, user_context)) async def process_async(experiment, result, user_context): # Send to your analytics system await analytics.track_async( user_id=user_context.attributes.get('id'), event_name="Experiment Viewed", properties={ 'experiment_id': experiment.key, 'variation_id': result.key, 'variation_value': result.value, } ) async def main(): # Create client with built-in tracking plugin client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", tracking_plugins=[ GrowthBookTrackingPlugin( # Ingestor host for GrowthBook built-in usage ingestor_host="https://c1.growthbook.io", # Additional callback for custom analytics additional_callback=my_custom_tracker ) ] ) ) try: await client.initialize() user = UserContext(attributes={"id": "user_123"}) # Events are now automatically tracked and batched result1 = await client.get_feature_value("button-color", "blue", user) result2 = await client.is_on("premium-feature", user) # Plugin handles batching and sending automatically finally: await client.close() asyncio.run(main()) ``` ```python from growthbook import GrowthBook, TrackingPlugin # Custom tracking callback for your analytics system def my_custom_tracker(events): """Process a batch of tracking events""" for event in events: # Send to your analytics system analytics.track( user_id=event.user_id, event_name="Experiment Viewed", properties={ 'experiment_id': event.experiment.key, 'variation_id': event.result.key, 'variation_value': event.result.value, 'timestamp': event.timestamp } ) print(f"Processed {len(events)} tracking events") # Create GrowthBook with tracking plugin gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={"id": "user_123"}, tracking_plugins=[ TrackingPlugin( # Your custom tracking function callback=my_custom_tracker, # Batch size (optional, default 10) batch_size=5, # Flush interval in seconds (optional, default 30) flush_interval=10, # Max retries (optional, default 3) max_retries=2 ) ] ) gb.load_features() # Events are now automatically tracked and batched result1 = gb.get_feature_value("button-color", "blue") result2 = gb.is_on("premium-feature") # Plugin handles batching and sending automatically gb.destroy() ``` #### Multiple Tracking Plugins You can use multiple tracking plugins to send events to different analytics systems: ```python from growthbook import GrowthBookClient, Options, GrowthBookTrackingPlugin # Different tracking callbacks def segment_tracker(experiment, result, user_context): """Send events to Segment (Fire-and-Forget)""" loop = asyncio.get_running_loop() loop.create_task(segment_track_async(experiment, result)) async def segment_track_async(experiment, result): await segment.track_async(...) async def mixpanel_tracker(events): """Send events to Mixpanel""" for event in events: await mixpanel.track_async(event.user_id, "Experiment Viewed", { 'experiment_id': event.experiment.key, 'variation': event.result.value }) # Create client with multiple tracking plugins client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", tracking_plugins=[ GrowthBookTrackingPlugin( additional_callback=segment_tracker ), GrowthBookTrackingPlugin( additional_callback=mixpanel_tracker ) ] ) ) # Events are now automatically sent to both Segment and Mixpanel ``` #### Tracking Plugin Benefits The tracking plugin system provides several advantages over custom tracking callbacks: - **Automatic Batching**: Events are batched together to reduce API calls - **Error Handling**: Failed requests are automatically retried with exponential backoff - **Non-blocking**: Tracking doesn't block feature evaluations - **Configurable**: Batch sizes, intervals, and retry logic can be customized - **Multiple Destinations**: Send events to multiple analytics systems simultaneously #### Working with Traditional Callbacks Tracking plugins work alongside your existing `on_experiment_viewed` callbacks: ```python def legacy_tracker(experiment, result): """Traditional tracking callback""" print(f"Legacy tracker: {experiment.key}") client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # Traditional callback still works on_experiment_viewed=legacy_tracker, # Plus new tracking plugins tracking_plugins=[ TrackingPlugin(callback=my_custom_tracker) ] ) ) # Both the legacy callback and plugin will be triggered ``` ## Using Features There are 3 main methods for interacting with features. - `gb.is_on("feature-key")` returns true if the feature is on - `gb.is_off("feature-key")` returns false if the feature is on - `gb.get_feature_value("feature-key", "default")` returns the value of the feature with a fallback In addition, you can use `gb.evalFeature("feature-key")` to get back a `FeatureResult` object with the following properties: - **value** - The JSON-decoded value of the feature (or `None` if not defined) - **on** and **off** - The JSON-decoded value cast to booleans - **source** - Why the value was assigned to the user. One of `unknownFeature`, `defaultValue`, `force`, or `experiment` - **experiment** - Information about the experiment (if any) which was used to assign the value to the user - **experimentResult** - The result of the experiment (if any) which was used to assign the value to the user ## Sticky Bucketing **Available starting in version 1.1.0** By default GrowthBook does not persist assigned experiment variations for a user. We rely on deterministic hashing to ensure that the same user attributes always map to the same experiment variation. However, there are cases where this isn't good enough. For example, if you change targeting conditions in the middle of an experiment, users may stop being shown a variation even if they were previously bucketed into it. Sticky Bucketing is a solution to these issues. You can provide a Sticky Bucket Service to the GrowthBook instance to persist previously seen variations and ensure that the user experience remains consistent for your users. A sample `InMemoryStickyBucketService` implementation is provided for reference, but in production you will definitely want to implement your own version using a database, cookies, or similar for persistence. Sticky Bucket documents contain three fields - `attributeName` - The name of the attribute used to identify the user (e.g. `id`, `cookie_id`, etc.) - `attributeValue` - The value of the attribute (e.g. `123`) - `assignments` - A dictionary of persisted experiment assignments. For example: `{"exp1__0":"control"}` The attributeName/attributeValue combo is the primary key. **Note**: The Async Client currently uses the synchronous sticky bucket interface. This means sticky bucket operations running in the main event loop may block. We recommend using a fast, local store (like the default in-memory one) or an optimized synchronous store. ```python import asyncio from growthbook import GrowthBookClient, Options, UserContext # AbstractStickyBucketService is synchronous from growthbook import AbstractStickyBucketService class MyStickyBucketService(AbstractStickyBucketService): def get_assignments(self, attributeName, attributeValue): # Implementation must be synchronous return None def save_assignments(self, doc): # Implementation must be synchronous pass async def main(): # Create sticky bucket service sticky_service = MyStickyBucketService() # Create client with sticky bucketing client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", sticky_bucket_service=sticky_service ) ) try: await client.initialize() # User will get consistent experiment assignments user = UserContext(attributes={"id": "user_123"}) # ... finally: await client.close() ``` For the traditional client, implement a synchronous sticky bucket service: ```python from typing import Optional, Dict from growthbook import GrowthBook, AbstractStickyBucketService class MyStickyBucketService(AbstractStickyBucketService): """Synchronous sticky bucket service using database""" def __init__(self): # Initialize your database connection self.db = Database() # Your DB client def get_assignments(self, attribute_name: str, attribute_value: str) -> Optional[Dict]: """Lookup a sticky bucket document""" try: return self.db.find({ "attributeName": attribute_name, "attributeValue": attribute_value }) except Exception as e: print(f"Error getting assignments: {e}") return None def save_assignments(self, doc: Dict) -> None: """Save sticky bucket assignments""" try: self.db.upsert({ "attributeName": doc["attributeName"], "attributeValue": doc["attributeValue"] }, { "$set": { "assignments": doc["assignments"] } }) except Exception as e: print(f"Error saving assignments: {e}") # Create GrowthBook with sticky bucketing gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={"id": "user_123"}, sticky_bucket_service=MyStickyBucketService() ) gb.load_features() # User will get consistent experiment assignments result1 = gb.get_feature_value("button-color", "blue") print(f"First assignment: {result1}") # Second time - retrieves saved assignment result2 = gb.get_feature_value("button-color", "blue") print(f"Consistent assignment: {result2}") # Run inline experiments with sticky bucketing from growthbook import Experiment experiment = Experiment( key="pricing-test", variations=["$9.99", "$14.99", "$19.99"] ) exp_result = gb.run(experiment) print(f"Sticky experiment result: {exp_result.value}") gb.destroy() ``` #### SQLite Implementation Example ```python import sqlite3 import json from growthbook import GrowthBook, AbstractStickyBucketService class SQLiteStickyBucketService(AbstractStickyBucketService): def __init__(self, db_path="sticky_buckets.db"): self.conn = sqlite3.connect(db_path) self.conn.execute(''' CREATE TABLE IF NOT EXISTS sticky_buckets ( attribute_name TEXT, attribute_value TEXT, assignments TEXT, PRIMARY KEY (attribute_name, attribute_value) ) ''') self.conn.commit() def get_assignments(self, attribute_name: str, attribute_value: str) -> Optional[Dict]: cursor = self.conn.execute( "SELECT assignments FROM sticky_buckets WHERE attribute_name=? AND attribute_value=?", (attribute_name, attribute_value) ) row = cursor.fetchone() if row: return { "attributeName": attribute_name, "attributeValue": attribute_value, "assignments": json.loads(row[0]) } return None def save_assignments(self, doc: Dict) -> None: self.conn.execute( "INSERT OR REPLACE INTO sticky_buckets (attribute_name, attribute_value, assignments) VALUES (?, ?, ?)", (doc["attributeName"], doc["attributeValue"], json.dumps(doc["assignments"])) ) self.conn.commit() # Usage gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={"id": "user_123"}, sticky_bucket_service=SQLiteStickyBucketService() ) ``` ## Inline Experiments Instead of declaring all features up-front and referencing them by ids in your code, you can also just run an experiment directly. This is done with the `run` method: For the async client, use `await` with the `run` method: ```python import asyncio from growthbook import GrowthBookClient, Options, UserContext, Experiment async def main(): client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" ) ) try: await client.initialize() # Create user context user = UserContext(attributes={"id": "user_123", "country": "US"}) # Simple experiment exp = Experiment( key="my-experiment", variations=["red", "blue", "green"] ) # Run experiment (async) result = await client.run(exp, user) print(f"Variation: {result.value}") # Either "red", "blue", or "green" # Complex experiment with all options complex_exp = Experiment( key="pricing-test", variations=[9.99, 14.99, 19.99], weights=[0.5, 0.3, 0.2], # 50%, 30%, 20% coverage=0.8, # Include 80% of users condition={"country": "US", "premium": True}, hashAttribute="id", hashVersion=2, namespace=("pricing", 0, 0.5) ) pricing_result = await client.run(complex_exp, user) if pricing_result.in_experiment: print(f"User is in pricing experiment: ${pricing_result.value}") print(f"Variation ID: {pricing_result.key}") print(f"Hash used: {pricing_result.hash_used}") else: print("User not in pricing experiment") finally: await client.close() asyncio.run(main()) ``` For the traditional client, call `run` directly: ```python from growthbook import GrowthBook, Experiment # Create GrowthBook instance gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={"id": "user_123", "country": "US"} ) gb.load_features() # Simple experiment exp = Experiment( key="my-experiment", variations=["red", "blue", "green"] ) # Run experiment result = gb.run(exp) print(f"Variation: {result.value}") # Either "red", "blue", or "green" # Complex experiment with all options complex_exp = Experiment( key="pricing-test", variations=[9.99, 14.99, 19.99], weights=[0.5, 0.3, 0.2], # 50%, 30%, 20% coverage=0.8, # Include 80% of users condition={"country": "US", "premium": True}, hashAttribute="id", hashVersion=2, namespace=("pricing", 0, 0.5) ) pricing_result = gb.run(complex_exp) if pricing_result.in_experiment: print(f"User is in pricing experiment: ${pricing_result.value}") print(f"Variation ID: {pricing_result.key}") print(f"Hash used: {pricing_result.hash_used}") else: print("User not in pricing experiment") gb.destroy() ``` As you can see, there are 2 required parameters for experiments, a string key, and an array of variations. Variations can be any data type, not just strings. There are a number of additional settings to control the experiment behavior: - **key** (`str`) - The globally unique tracking key for the experiment - **variations** (`any[]`) - The different variations to choose between - **seed** (`str`) - Added to the user id when hashing to determine a variation. Defaults to the experiment `key` - **weights** (`float[]`) - How to weight traffic between variations. Must add to 1. - **coverage** (`float`) - What percent of users should be included in the experiment (between 0 and 1, inclusive) - **condition** (`dict`) - Targeting conditions - **force** (`int`) - All users included in the experiment will be forced into the specified variation index - **hashAttribute** (`string`) - What user attribute should be used to assign variations (defaults to "id") - **hashVersion** (`int`) - What version of our hashing algorithm to use. We recommend using the latest version `2`. - **namespace** (`tuple[str,float,float]`) - Used to run mutually exclusive experiments. Here's an example that uses all of them: ```python exp = Experiment( key="my-test", # Variations can be a list of any data type variations=[0, 1], # If this changes, it will re-randomize all users in the experiment seed="abcdef123456", # Run a 40/60 experiment instead of the default even split (50/50) weights=[0.4, 0.6], # Only include 20% of users in the experiment coverage=0.2, # Targeting condition using a MongoDB-like syntax condition={ 'country': 'US', 'browser': { '$in': ['chrome', 'firefox'] } }, # Use an alternate attribute for assigning variations (default is 'id') hashAttribute="sessionId", # Use the latest hashing algorithm hashVersion=2, # Includes the first 50% of users in the "pricing" namespace # Another experiment with a non-overlapping range will be mutually exclusive (e.g. [0.5, 1]) namespace=("pricing", 0, 0.5), ) ``` ### Inline Experiment Return Value A call to `run` returns a `Result` object with a few useful properties: ```python result = gb.run(exp) # If user is part of the experiment print(result.inExperiment) # True or False # The string key of the assigned variation print(result.key) # e.g. "0" or "1" # The value of the assigned variation print(result.value) # e.g. "A" or "B" # If the variation was randomly assigned by hashing user attributes print(result.hashUsed) # True or False # The user attribute used to assign a variation print(result.hashAttribute) # "id" # The value of that attribute print(result.hashValue) # e.g. "123" ``` The `inExperiment` flag will be false if the user was excluded from being part of the experiment for any reason (e.g. failed targeting conditions). The `hashUsed` flag will only be true if the user was randomly assigned a variation. If the user was forced into a specific variation instead, this flag will be false. ### Example Experiments 3-way experiment with uneven variation weights: ```python result = await client.run( Experiment( key="3-way-uneven", variations=["A", "B", "C"], weights=[0.5, 0.25, 0.25] ), user ) ``` 3-way experiment with uneven variation weights: ```python gb.run(Experiment( key="3-way-uneven", variations=["A", "B", "C"], weights=[0.5, 0.25, 0.25] )) ``` Slow rollout (10% of users who match the targeting condition): ```python # Create user context with targeting attributes user = UserContext(attributes={ "id": "123", "beta": True, "qa": True, }) result = await client.run( Experiment( key="slow-rollout", variations=["A", "B"], coverage=0.1, condition={'beta': True} ), user ) ``` ```python # User is marked as being in "qa" and "beta" gb = GrowthBook( attributes={ "id": "123", "beta": True, "qa": True, }, ) gb.run(Experiment( key="slow-rollout", variations=["A", "B"], coverage=0.1, condition={'beta': True} )) ``` Complex variations: ```python result = await client.run( Experiment( key="complex-variations", variations=[ ("blue", "large"), ("green", "small") ], ), user ) # Either "blue,large" OR "green,small" print(result.value[0] + "," + result.value[1]) ``` ```python result = gb.run(Experiment( key="complex-variations", variations=[ ("blue", "large"), ("green", "small") ], )) # Either "blue,large" OR "green,small" print(result.value[0] + "," + result.value[1]) ``` Assign variations based on something other than user id: ```python # Create user context with company attribute user = UserContext(attributes={ "id": "123", "company": "growthbook" }) # Users in the same company will always get the same variation result = await client.run( Experiment( key="by-company-id", variations=["A", "B"], hashAttribute="company" ), user ) ``` ```python gb = GrowthBook( attributes={ "id": "123", "company": "growthbook" } ) # Users in the same company will always get the same variation gb.run(Experiment( key="by-company-id", variations=["A", "B"], hashAttribute="company" )) ``` ## Working with Encrypted Features The Python SDK supports encrypted feature flags for enhanced security. When encryption is enabled, the feature payload is encrypted before being sent from GrowthBook, and the SDK automatically decrypts it client-side. For the async client, provide the decryption key in the `Options`: ```python import asyncio from growthbook import GrowthBookClient, Options, UserContext async def main(): # Create client with decryption key client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # Your decryption key from GrowthBook decryption_key="your-secret-key-here" ) ) try: await client.initialize() # Features are automatically decrypted user = UserContext(attributes={"id": "user_123"}) feature_enabled = await client.is_on("encrypted-feature", user) print(f"Encrypted feature enabled: {feature_enabled}") finally: await client.close() asyncio.run(main()) ``` For the traditional client, pass the decryption key to the constructor: ```python from growthbook import GrowthBook # Create GrowthBook with decryption key gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # Your decryption key from GrowthBook decryption_key="your-secret-key-here", attributes={"id": "user_123"} ) gb.load_features() # Features are automatically decrypted feature_enabled = gb.is_on("encrypted-feature") feature_value = gb.get_feature_value("encrypted-config", {"default": "value"}) print(f"Encrypted feature enabled: {feature_enabled}") print(f"Encrypted config: {feature_value}") gb.destroy() ``` ### Environment Variables You can also set the decryption key via environment variable: ```bash export GROWTHBOOK_DECRYPTION_KEY="your-secret-key-here" ``` ```python import os from growthbook import GrowthBookClient, Options # Decryption key will be automatically picked up from environment client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", decryption_key=os.environ.get("GROWTHBOOK_DECRYPTION_KEY") ) ) ``` ```python import os from growthbook import GrowthBook # Decryption key will be automatically picked up from environment gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", # decryption_key will be read from GROWTHBOOK_DECRYPTION_KEY env var attributes={"id": "user_123"} ) ``` ### Error Handling If decryption fails (wrong key, corrupted data, etc.), the SDK will log an error and treat all features as disabled/default values: ```python import logging # Enable logging to see decryption errors logging.basicConfig(level=logging.ERROR) logger = logging.getLogger('growthbook') # This will log errors if decryption fails gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", decryption_key="wrong-key", # This will cause decryption to fail attributes={"id": "user_123"} ) ``` ## Logging The GrowthBook SDK uses a Python logger with the name `growthbook` and includes helpful info for debugging as well as warnings/errors if something is misconfigured. Here's an example of logging to the console ```python import logging logger = logging.getLogger('growthbook') logger.setLevel(logging.DEBUG) handler = logging.StreamHandler() formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s %(message)s') handler.setFormatter(formatter) logger.addHandler(handler) ``` ## Integration Examples This section provides practical examples for integrating GrowthBook with popular web frameworks. ### Async Web Framework Integration (FastAPI) The async client works great with modern async web frameworks like FastAPI: ```python from fastapi import FastAPI, Depends from growthbook import GrowthBookClient, Options, UserContext app = FastAPI() # Create a single client instance (singleton) gb_client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" ) ) @app.on_event("startup") async def startup(): # Initialize the client when the app starts await gb_client.initialize() @app.on_event("shutdown") async def shutdown(): # Clean up when the app shuts down await gb_client.close() @app.get("/") async def root(user_id: str): # Create user context for the request user = UserContext(attributes={"id": user_id}) # Use features - no need to pass client around show_new_ui = await gb_client.is_on("new-ui", user) discount_amount = await gb_client.get_feature_value("discount-percent", 0, user) return { "new_ui": show_new_ui, "discount": discount_amount } @app.get("/experiment") async def run_experiment(user_id: str): user = UserContext(attributes={"id": user_id}) # Run inline experiments from growthbook import Experiment experiment = Experiment( key="button-color", variations=["red", "blue", "green"] ) result = await gb_client.run(experiment, user) return { "variation": result.value, "in_experiment": result.in_experiment } ``` ### Starlette Integration ```python from starlette.applications import Starlette from starlette.responses import JSONResponse from starlette.routing import Route from growthbook import GrowthBookClient, Options, UserContext # Create client instance gb_client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123" ) ) async def homepage(request): user_id = request.query_params.get('user_id', 'anonymous') user = UserContext(attributes={"id": user_id}) feature_enabled = await gb_client.is_on("new-homepage", user) return JSONResponse({ "new_homepage": feature_enabled, "user_id": user_id }) async def startup(): await gb_client.initialize() async def shutdown(): await gb_client.close() app = Starlette( routes=[ Route('/', homepage), ], on_startup=[startup], on_shutdown=[shutdown] ) ``` ### Traditional Web Frameworks (Django, Flask, etc.) **For new projects, we recommend using the [Async Client](#async-client-recommended) instead for better performance.** For traditional synchronous web frameworks, you should create a new `GrowthBook` instance for every incoming request and call `destroy()` at the end of the request to clean up resources. #### Django Integration In Django, this is best done with a simple middleware: ```python from growthbook import GrowthBook def growthbook_middleware(get_response): def middleware(request): request.gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={ "id": getattr(request.user, 'id', 'anonymous'), "loggedIn": request.user.is_authenticated, "country": request.META.get('HTTP_CF_IPCOUNTRY', 'US') } ) request.gb.load_features() response = get_response(request) request.gb.destroy() # Cleanup return response return middleware ``` Then, you can easily use GrowthBook in any of your views: ```python def index(request): feature_enabled = request.gb.is_on("my-feature") button_color = request.gb.get_feature_value("button-color", "blue") return render(request, 'index.html', { 'feature_enabled': feature_enabled, 'button_color': button_color }) ``` #### Flask Integration ```python from flask import Flask, g, request from growthbook import GrowthBook app = Flask(__name__) @app.before_request def before_request(): g.gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={ "id": request.headers.get('User-ID', 'anonymous'), "userAgent": request.user_agent.string, "url": request.url } ) g.gb.load_features() @app.teardown_request def teardown_request(exception): gb = getattr(g, 'gb', None) if gb is not None: gb.destroy() @app.route('/') def index(): feature_enabled = g.gb.is_on("new-layout") theme = g.gb.get_feature_value("theme", "light") return { "feature_enabled": feature_enabled, "theme": theme } ``` ## Supported Features --- ## SDK Quickstart import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import { CodeLogos } from '@site/src/components/Icons'; import ButtonCard from '@site/src/components/ButtonCard'; import { Stepper, StepperStep } from '@site/src/components/Stepper'; # SDK Quickstart Get GrowthBook running in your application in minutes. :::info New to GrowthBook SDKs? The [SDK Overview](/lib) covers how SDKs work and helps you choose the right one for your use case. ::: **Get your SDK Client Key** Go to [**SDK Configuration**](https://app.growthbook.io/sdks), create a new SDK Connection, and copy the **Client Key** (starts with `sdk-`). **Install the SDK** ```bash npm2yarn npm install @growthbook/growthbook-react @growthbook/growthbook ``` **Wrap your app with `GrowthBookProvider`** ```tsx import { GrowthBook, GrowthBookProvider } from "@growthbook/growthbook-react"; import { thirdPartyTrackingPlugin, autoAttributesPlugin } from "@growthbook/growthbook/plugins"; // Create a GrowthBook instance const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", // Your SDK client key enableDevMode: true, plugins: [ thirdPartyTrackingPlugin(), // Optional, sends "Experiment Viewed" events via GrowthBook Managed Warehouse, Google Analytics, Google Tag Manager, and Segment. autoAttributesPlugin(), // Optional, sets common attributes (browser, session_id, etc.) ], }); // Load feature definitions from the GrowthBook API gb.init(); export default function App() { return ( ); } ``` **Use feature flags** ```tsx import { useFeatureIsOn, useFeatureValue } from "@growthbook/growthbook-react"; function MyApp() { const showNewFeature = useFeatureIsOn("new-feature"); const buttonColor = useFeatureValue("button-color", "blue"); return ( {showNewFeature && } Click me ); } ``` **Next steps:** - See the [example Next.js app](https://github.com/growthbook/examples/tree/main/next-js) - See the [React SDK docs](/lib/react) **Get your SDK Client Key** Go to [**SDK Configuration**](https://app.growthbook.io/sdks), create a new SDK Connection, and copy the **Client Key** (starts with `sdk-`). **Install the SDK** ```bash npm2yarn npm install @growthbook/growthbook ``` **Initialize GrowthBook** ```javascript import { GrowthBook } from "@growthbook/growthbook"; import { thirdPartyTrackingPlugin, autoAttributesPlugin } from "@growthbook/growthbook/plugins"; const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", plugins: [ thirdPartyTrackingPlugin(), // Optional, sends "Experiment Viewed" events via GrowthBook Managed Warehouse, Google Analytics, Google Tag Manager, and Segment. autoAttributesPlugin(), // Optional, sets common attributes (browser, session_id, etc.) ], }); await gb.init(); ``` **Use feature flags** ```javascript if (gb.isOn("new-feature")) { showNewFeature(); } const color = gb.getFeatureValue("button-color", "blue"); ``` **Next steps:** - See the [JavaScript SDK docs](/lib/js) for TypeScript, plugins, and more. - See an [example Typescript app](https://github.com/growthbook/examples/tree/main/vanilla-typescript) **Get your SDK Client Key** Go to [**SDK Configuration**](https://app.growthbook.io/sdks), create a new SDK Connection, and copy the **Client Key** (starts with `sdk-`). **Install the SDK** ```bash npm2yarn npm install @growthbook/growthbook ``` **Initialize GrowthBook** ```javascript import { GrowthBookClient } from "@growthbook/growthbook"; const gbClient = new GrowthBookClient({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", trackingCallback: (experiment, result, userContext) => { const userId = userContext.attributes.id; // Required for A/B testing // TODO: Replace with your own tracking implementation console.log("Viewed Experiment", userId, { experimentId: experiment.key, variationId: result.key, attributes: userContext.attributes, }); } }); await gbClient.init({timeout: 3000}); ``` **Use feature flags** ```javascript // User context with attributes that feature flags can use for targeting const userContext = { attributes: { id: "123", country: "US", // ... other attributes ... } } // Boolean on/off flags if (gbClient.isOn("my-feature", userContext)) { console.log("My feature is on!"); } // String, Number, or JSON flags const value = gbClient.getFeatureValue("my-string-feature", "fallback", userContext); console.log(value); ``` **Next steps:** - See the full [Node.js SDK docs](/lib/node) - Use the SDK in [Express middleware](/lib/node#express-middleware) **Get your SDK Client Key** Go to [**SDK Configuration**](https://app.growthbook.io/sdks), create a new SDK Connection, and copy the **Client Key** (starts with `sdk-`). **Install the SDK** ```bash pip pip install growthbook ``` **Initialize GrowthBook** ```python from growthbook import GrowthBookClient, Options, UserContext def on_experiment_viewed(experiment, result): # TODO: track in your analytics system print(f"Experiment: {experiment.key}, Variation: {result.key}") # Create and initialize client client = GrowthBookClient( Options( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", on_experiment_viewed=on_experiment_viewed, ) ) await client.initialize() # Create user context for targeting user = UserContext( attributes={ "id": "user-123", "country": "US", } ) ``` **Use feature flags** ```python if await client.is_on("new-feature", user): show_new_feature() color = await client.get_feature_value("button-color", "blue", user) ``` **Next steps:** - See the full [Python SDK docs](/lib/python) - See the [how to integrate with FastAPI](/lib/python#async-web-framework-integration-fastapi) example. ## All SDKs **Client** } /> } /> } /> } /> } /> } /> } /> } /> **Server** } /> } /> } /> } /> } /> } /> } /> } /> } /> **Edge** } /> } /> } /> --- ## Verify your setup After setting up your SDK, verify it's working by following these steps: 1. Create a feature flag called `test-flag` in your [GrowthBook](https://app.growthbook.io/features) 2. Set its value to `true` and enable it for your environment 3. Check that `isOn("test-flag")` returns `true` in your app 4. Toggle the flag off and verify the change is reflected :::tip Use the [GrowthBook DevTools](/tools/chrome-extension) browser extension to inspect and test feature flags in development. ::: --- ## React Native SDK import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # React Native This is a thin wrapper on top of the [Javascript Library](/lib/js), so you might want to view those docs first to familiarize yourself with the basic classes and methods. ## Installation Install with a package manager ```bash npm2yarn npm install --save @growthbook/growthbook-react ``` ## Quick Usage ### Step 1: Configure your app ```tsx import { useEffect } from "react"; import { GrowthBook, GrowthBookProvider } from "@growthbook/growthbook-react"; // Create a GrowthBook instance const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", // Only required for A/B testing // Called every time a user is put into an experiment trackingCallback: (experiment, result, user) => { console.log("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, attributes: user?.attributes, }); }, }); gb.init() export default function App() { useEffect(() => { // TODO: Set user attributes for targeting gb.setAttributes({ id: user.id, company: user.company, }); }, [user]) return ( ); } ``` ### Step 2: Start feature flagging! There are a few ways to use feature flags in GrowthBook: #### Feature Hooks ```tsx import { useFeatureValue, useFeatureIsOn } from "@growthbook/growthbook-react"; export default function OtherComponent() { // Boolean on/off features const newLogin = useFeatureIsOn("new-login-form"); // String/Number/JSON features with a fallback value const buttonColor = useFeatureValue("login-button-color", "blue"); if (newLogin) { return ; } else { return ; } } ``` #### Feature Wrapper Components ```jsx import { IfFeatureEnabled, FeatureString } from "@growthbook/growthbook-react"; export default function OtherComponent() { return ( Welcome to our app! ); } ``` #### useGrowthBook hook If you need low-level access to the GrowthBook instance for any reason, you can use the `useGrowthBook` hook. One example is updating targeting attributes when a user logs in: ```jsx import { useGrowthBook } from "@growthbook/growthbook-react"; export default function Auth() { const growthbook = useGrowthBook(); const user = useUser(); useEffect(() => { if (!user) return; growthbook.setAttributes({ loggedIn: true, id: user.id, company: user.company, isPro: user.plan === "pro" }) }, [user, growthbook]) ... } ``` ## Loading Features In order for the GrowthBook SDK to work, it needs to have feature definitions from the GrowthBook API. There are 2 ways to get this data into the SDK. ### Built-in Fetching and Caching If you pass an `apiHost` and `clientKey` into the GrowthBook constructor, it will handle the network requests, caching, retry logic, etc. for you automatically. If your feature payload is encrypted, you can also pass in a `decryptionKey`. ```ts const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", }); await gb.init({ // If the network request takes longer than this (in milliseconds), continue // Default: `0` (no timeout) timeout: 2000, }) ``` Until features are loaded, all features will evaluate to `null`. If you're ok with a potential flicker in your application (features going from `null` to their real value), you can call `init` without awaiting the result. If you want to refresh the features at any time (e.g. when a navigation event occurs), you can call `gb.refreshFeatures()`. #### Error Handling In the case of network issues, the `init` call will not throw an error. Instead, it will stay in the default state where every feature evaluates to `null`. You can still get access to the error if needed: ```ts const res = await gb.init({ timeout: 1000 }); console.log(res); ``` The return value has 3 properties: - **status** - `true` if the GrowthBook instance was populated with features/experiments. Otherwise `false` - **source** - Where this result came from. One of the following values: `network`, `cache`, `init`, `error`, or `timeout` - **error** - If status is `false`, this will contain an `Error` object with more details about the error ### Custom Integration If you prefer to handle the network and caching logic yourself, you can pass in a full JSON "payload" directly into the SDK. For example, you might store features in Postgres and send it down as part of your app's initial bootstrap API call. ```ts await gb.init({ payload: { features: { "feature-1": {...}, "feature-2": {...}, "another-feature": {...}, } } }) ``` The data structure for "payload" is exactly the same as what is returned by the GrowthBook SDK endpoints and webhooks. You can update the payload at any time by calling `setPayload(newPayloadJSON)`. Note: you don't need to specify `clientKey` or `apiHost` on your GrowthBook instance unless you want to enable streaming (see below) or call `refreshFeatures()` later. #### Synchronous Init There is a alternate synchronous version of init named `initSync`, which can be especially useful in SSR to prevent hydration mismatches. There are some restrictions/differences: - You MUST pass in `payload` - The `payload` MUST NOT have encrypted features or experiments - If using sticky bucketing, you should use an instance of `StickyBucketServiceSync`. - The return value is the GrowthBook instance to enable easy method chaining ## Waiting for Features to Load There is a helper component `` that lets you render a fallback component until features are done loading. This works for both built-in fetching and custom integrations. ```jsx }> ``` - `timeout` is the max time you want to wait for features to load (in ms). The default is `0` (no timeout). - `fallback` is the component you want to display before features are loaded. The default is `null`. If you want more control, you can use the `useGrowthBook()` hook and the `ready` flag: ```ts const gb = useGrowthBook(); if (gb.ready) { // Do something } ``` ## Streaming Updates The GrowthBook SDK supports streaming with Server-Sent Events (SSE). When enabled, changes to features within GrowthBook will be streamed to the SDK in realtime as they are published. This is only supported on GrowthBook Cloud or if running a GrowthBook Proxy Server. React Native does not support SSE out-of-the-box, but there is a small helper library you can install: ```bash npm2yarn npm install --save react-native-sse ``` The, tell GrowthBook to use this polyfill: ```js import { setPolyfills } from "@growthbook/growthbook"; import EventSource from "react-native-sse"; // Configure GrowthBook to use the eventsource library setPolyfills({ EventSource: EventSource }); ``` And finally, you can simply pass `streaming: true` into your init calls: ```ts gb.init({ streaming: true, }) ``` ## Remote Evaluation The React Native SDK may be run in Remote Evaluation mode. This mode brings the security benefits of a backend SDK to mobile by evaluating feature flags exclusively on a private server. Using Remote Evaluation ensures that any sensitive information within targeting rules or unused feature variations are never seen by the client. You must enable Remote Evaluation in your SDK Connection settings. Cloud customers are also required to self-host a GrowthBook Proxy Server or custom remote evaluation backend. To use Remote Evaluation, add the `remoteEval: true` property to your SDK instance. A new evaluation API call will be made any time a user attribute or other dependency changes. You may optionally limit these API calls to specific attribute changes by setting the `cacheKeyAttributes` property (an array of attribute names that, when changed, trigger a new evaluation call). ```ts const gb = new GrowthBook({ apiHost: "https://gb-proxy.mydomain.io/", clientKey: "sdk-abc123", // Enable remote evaluation remoteEval: true, // Optional: only trigger a new evaluation call when the `id` and `email` attribute changes cacheKeyAttributes: ["id", "email"], }); ``` :::note Sticky Bucketing with Remote Evaluation If you would like to implement Sticky Bucketing while using Remote Evaluation, you must configure your remote evaluation backend to support Sticky Bucketing. In the case of the GrowthBook Proxy Server, this means implementing a Redis database for sticky bucketing use. You will not need to provide a StickyBucketService instance to the client side SDK. ::: ## Caching The React Native SDK has 2 caching layers: 1. In-memory cache (enabled by default) 2. Persistent localStorage cache (disabled by default, requires configuration) ### Configuring Local Storage In order to use persistent storage, you must provide a polyfill with the same method signature as browser's localStorage. Here's an example using AsyncStorage ```bash npm2yarn npm install --save @react-native-async-storage/async-storage ``` ```jsx import AsyncStorage from '@react-native-async-storage/async-storage'; import { setPolyfills } from "@growthbook/growthbook"; setPolyfills({ localStorage: { // Example using Redis getItem: (key) => JSON.parse(AsyncStorage.getItem(key) || "null") setItem: (key, value) => AsyncStorage.setItem(key, JSON.stringify(value)) } }); ``` ### Cache Settings There are a number of cache settings you can configure within GrowthBook. This must be done BEFORE creating a GrowthBook instance. Below are all of the default values. You can call `configureCache` with a subset of these fields and the rest will keep their default values. ```ts import { configureCache } from "@growthbook/growthbook"; configureCache({ // The localStorage key the cache will be stored under cacheKey: "gbFeaturesCache", // Consider features stale after this much time (60 seconds default) staleTTL: 1000 * 60, // Cached features older than this will be ignored (24 hours default) maxAge: 1000 * 60 * 60 * 24, // For Remote Eval only - limit the number of cache entries (~1 entry per user) maxEntries: 10, // Set to `true` to completely disable both in-memory and persistent caching disableCache: false, }) ``` ## Experimentation (A/B Testing) In order to run A/B tests, you need to set up a tracking callback function. This is called every time a user is put into an experiment and can be used to track the exposure event in your analytics system (Segment, Mixpanel, GA, etc.). ```js const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", trackingCallback: (experiment, result, user) => { // Example using Segment analytics.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, attributes: user?.attributes, }); }, }); ``` Starting in version 1.7.0, the callback receives a third `user` argument: a `TrackingUserContext` containing the `attributes` and optional `url` used when the experiment was evaluated. We recommend recording attributes from this argument so exposure events consistently reflect the attributes GrowthBook used for targeting and assignment. This same tracking callback is used for both feature flag experiments and Visual Editor experiments. ### Feature Flag Experiments There is nothing special you have to do for feature flag experiments. Just evaluate the feature flag like you would normally do. If the user is put into an experiment as part of the feature flag, it will call the `trackingCallback` automatically in the background. ```js // If this has an active experiment and the user is included, // it will call trackingCallback automatically useFeatureIsOn("new-signup-form") ``` If the experiment came from a feature rule, `result.featureId` in the trackingCallback will contain the feature id, which may be useful for tracking/logging purposes. ### Visual Editor Experiments Visual Editor experiments are not supported in React Native at this time. ### URL Redirect Experiments URL Redirect experiments are no supported in React Native at this time. ### Sticky Bucketing Sticky bucketing ensures that users see the same experiment variant, even when user session, user login status, or experiment parameters change. See the [Sticky Bucketing docs](/app/sticky-bucketing) for more information. If your organization and experiment supports sticky bucketing, you must implement an instance of the `StickyBucketService` to use Sticky Bucketing. The JS SDK exports several implementations of this service for reference, although you will need to build your own to work in a React Native environments. Implement the abstract `StickyBucketService` class and connect to your own data store, or custom wrap multiple service implementations. ```js import Cookies from 'js-cookie'; const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", stickyBucketService: new MyStickyBucketService(), // ... }); ``` ## API Reference There are a number of configuration options and settings that control how GrowthBook behaves. ### Attributes You can specify attributes about the current user and request. These are used for two things: 1. Feature targeting (e.g. paid users get one value, free users get another) 2. Assigning persistent variations in A/B tests (e.g. user id "123" always gets variation B) The following are some commonly used attributes, but use whatever makes sense for your application. ```ts new GrowthBook({ attributes: { id: "123", loggedIn: true, deviceId: "abc123def456", company: "acme", paid: false, url: "/pricing", browser: "chrome", mobile: false, country: "US", }, }); ``` #### Updating Attributes If attributes change, you can call `setAttributes()` to update. This will completely overwrite any existing attributes. To do a partial update, use the following pattern: ```js gb.setAttributes({ // Only update the `url` attribute, keep the rest the same ...gb.getAttributes(), url: "/new-page" }) ``` #### Secure Attributes When _secure attribute hashing_ is enabled, all targeting conditions in the SDK payload referencing attributes with datatype `secureString` or `secureString[]` will be anonymized via SHA-256 hashing. This allows you to safely target users based on sensitive attributes. You must enable this feature in your SDK Connection for it to take effect. If your SDK Connection has secure attribute hashing enabled, you will need to manually hash any `secureString` or `secureString[]` attributes that you pass into the GrowthBook SDK. To hash an attribute, use a cryptographic library with SHA-256 support, and compute the SHA-256 hashed value of your attribute _plus_ your organization's secure attribute salt. ```js const salt = "f09jq3fij"; // Your organization's secure attribute salt (see Organization Settings) // hashing a secureString attribute const userEmail = sha256(salt + user.email); // hashing an secureString[] attribute const userTags = user.tags.map(tag => sha256(salt + tag)); gb.setAttributes({ id: user.id, loggedIn: true, email: userEmail, tags: userTags, }); await gb.init(); // In this example, we are using Node.js's built-in crypto library function sha256(str) { return crypto.createHash("sha256").update(str).digest("hex"); } ``` Note that in a browser context, we will not be able to natively access the Node.js crypto library. In modern browsers `window.crypto.subtle` is available, although calls are asynchronous. You would need to await all attribute hashing to complete before calling `gb.setAttributes()`. ```js async function sha256(str) { const buffer = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(str)); const hashArray = Array.from(new Uint8Array(buffer)); return hashArray.map(byte => byte.toString(16).padStart(2, "0")).join(""); } ``` Alternatively, CryptoJS (https://www.npmjs.com/package/crypto-js) provides a synchronous API: ```js import sha256 from 'crypto-js/sha256'; const userEmail = sha256(salt + user.email).toString(); ``` ### Feature Usage Callback GrowthBook can fire a callback whenever a feature is evaluated for a user. This can be useful to update 3rd party tools like NewRelic or DataDog. ```ts new GrowthBook({ onFeatureUsage: (featureKey, result) => { console.log("feature", featureKey, "has value", result.value); }, }); ``` Note: If you evaluate the same feature multiple times (and the value doesn't change), the callback will only be fired the first time. ### Inline Experiments Depending on how you configure feature flags, they may run A/B tests behind the scenes to determine which value gets assigned to the user. Sometimes though, you want to run an inline experiment without going through a feature flag first. For this, you can use either the `useExperiment` hook or the Higher Order Component `withRunExperiment`: View the [Javascript SDK Docs](/lib/js) for all of the options available for inline experiments #### useExperiment hook ```tsx import { useExperiment } from "@growthbook/growthbook-react"; export default function OtherComponent() { const { value } = useExperiment({ key: "new-headline", variations: ["Hello", "Hi", "Good Day"] }); return {value}; } ``` #### withRunExperiment (class components) **Note:** This library uses hooks internally, so still requires React 16.8 or above. ```tsx import { withRunExperiment } from "@growthbook/growthbook-react"; class OtherComponent extends React.Component { render() { // The `runExperiment` prop is identical to the `useExperiment` hook const { value } = this.props.runExperiment({ key: "headline-test", variations: ["Hello World", "Hola Mundo"] }); return {value}; } } // Wrap your component in `withRunExperiment` export default withRunExperiment(OtherComponent); ``` ## TypeScript support Some hooks are available in type-safe versions. These require you to pass in your generated types as the generic argument. See the [GrowthBook CLI](/tools/cli) documentation for more information on generating type definitions and [JavaScript → TypeScript → Strict Typing](/lib/js#strict-typing) for how to use them. ### useGrowthBook<T>() A type-safe version of the `useGrowthBook()` hook is available. Everywhere you use `useGrowthBook()`, pass the generated features as the generic argument: ```ts const growthbook = useGrowthBook() ``` In that case, the hook will return `GrowthBook | undefined`. You can reduce this boilerplate by creating your own hook, e.g.: ```ts // ./src/utils/growthbook.ts import { useGrowthBook as _useGrowthBook } from "@growthbook/growthbook-react"; export const useGrowthBook = (): GrowthBook | undefined => _useGrowthBook(); ``` You can now reference the hook you created instead of the one from the official package: ```ts import { useGrowthBook } from "@/src/utils/growthbook" const growthbook = useGrowthBook(); growthbook.getFeatureValue(knownKey, defaultValueOfValidType) ``` ### useFeatureIsOn<T>() The React SDK also provides access to a type-safe `useFeatureIsOn()` hook. ```ts const isDarkModeOn = useFeatureIsOn("dark_mode"); ``` This will only allow you to pass known keys to the hook. You can reduce the boilerplate for this hook by creating your own and using that instead: ```ts // ./src/utils/growthbook.ts import { useFeatureIsOn as _useFeatureIsOn } from "@growthbook/growthbook-react"; export const useFeatureIsOn = (id: keyof AppFeatures & string): boolean => _useFeatureIsOn(id); ``` And then reference the hook you created instead of the one from the official package: ```ts import { useFeatureIsOn } from "@/src/utils/growthbook" const isDarkModeOn = useFeatureIsOn("dark_mode"); ``` ## Examples - [React Native ](https://github.com/growthbook/examples/tree/main/react-native-cli) - [Typescript example app with strict typing ](https://github.com/growthbook/examples/tree/main/vanilla-typescript). ## Supported Features --- ## React SDK import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # ReactJS This is a thin wrapper on top of the [Javascript Library](/lib/js), so you might want to view those docs first to familiarize yourself with the basic classes and methods. This SDK supports both ReactJS and ReactNative environments and both client and server components. **Important**: Starting in version 1.0.0, you must always pass a GrowthBook instance into the GrowthBookProvider. In previous versions, you were allowed to pass `null` as well. ## Installation Install with a package manager ```bash npm2yarn npm install --save @growthbook/growthbook-react ``` ## Quick Usage ### Step 1: Configure your app ```tsx import { useEffect } from "react"; import { GrowthBook, GrowthBookProvider } from "@growthbook/growthbook-react"; // Create a GrowthBook instance const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", enableDevMode: true, // Only required for A/B testing // Called every time a user is put into an experiment trackingCallback: (experiment, result, user) => { console.log("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, attributes: user?.attributes, }); }, }); gb.init({ // Optional, enable streaming updates streaming: true }) export default function App() { useEffect(() => { // Set user attributes for targeting (from cookie, auth system, etc.) gb.setAttributes({ id: user.id, company: user.company, }); }, [user]) return ( ); } ``` ### Step 2: Start feature flagging! There are a few ways to use feature flags in GrowthBook: #### Feature Hooks ```tsx import { useFeatureValue, useFeatureIsOn } from "@growthbook/growthbook-react"; export default function OtherComponent() { // Boolean on/off features const newLogin = useFeatureIsOn("new-login-form"); // String/Number/JSON features with a fallback value const buttonColor = useFeatureValue("login-button-color", "blue"); if (newLogin) { return ; } else { return ; } } ``` #### Feature Wrapper Components ```tsx import { IfFeatureEnabled, FeatureString } from "@growthbook/growthbook-react"; export default function OtherComponent() { return ( Welcome to our site! ); } ``` #### useGrowthBook hook If you need low-level access to the GrowthBook instance for any reason, you can use the `useGrowthBook` hook. One example is updating targeting attributes when a user logs in: ```jsx import { useGrowthBook } from "@growthbook/growthbook-react"; export default function Auth() { const growthbook = useGrowthBook(); const user = useUser(); useEffect(() => { if (!user) return; growthbook.setAttributes({ loggedIn: true, id: user.id, company: user.company, isPro: user.plan === "pro" }) }, [user, growthbook]) ... } ``` ## Plugins GrowthBook comes with a number of built-in plugins that add additional functionality. You can enable these by passing them into the GrowthBook constructor. Plugins require GrowthBook version 1.4.0 or higher, released in Feb 2025. ### Third Party Tracking The third-party tracking plugin automatically sends an "Experiment Viewed" event to several popular analytics tracking tools (Segment, Google Analytics, and Google Tag Manager). This plugin requires a browser environment and does not work in Server Components or React Native. This plugin does not install any 3rd party libraries, it just sends events to ones that are already installed on your site. Below are all of the supported trackers: - `segment` - Requires `window.analytics` to be defined. - `gtag` - Requires `window.gtag` to be defined. - `gtm` - Requires `window.dataLayer` to be defined. ```js import { thirdPartyTrackingPlugin } from "@growthbook/growthbook/plugins"; // Optional settings for the plugin const pluginOptions = { // By default, it will attempt to send to all 3 of these trackers: ["segment", "gtag", "gtm"], // Additional custom tracking callback additionalCallback: (experiment, result) => { console.log("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, }) } } const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", plugins: [ thirdPartyTrackingPlugin(pluginOptions), ], }); ``` ### Auto-Attributes The auto-attributes plugin automatically detects and sets common attributes in GrowthBook for the current user. These attributes are available locally for targeting and are not sent to GrowthBook's servers. This plugin requires a browser environment and does not work in Server Components or React Native. ```js import { autoAttributesPlugin } from "@growthbook/growthbook/plugins"; // Optional settings for the plugin (documented below) const pluginOptions: {} const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", plugins: [ autoAttributesPlugin(pluginOptions), ], }); ``` #### Attributes Here is a list of all attributes that are set by this plugin: - `id` - A random unique identifier for the user, persisted in a cookie - `url` - The current URL of the page - `path` - The current URL path of the page - `host` - The current URL host of the page - `query` - The current URL query string of the page - `pageTitle` - The current page title - `deviceType` - The device type of the user. Either "mobile" or "desktop" - `browser` - The browser name of the user (edge, chrome, firefox, safari, or unknown) - `utmSource`, `utmMedium`, `utmCampaign`, `utmTerm`, `utmContent` - UTM parameters from the URL (persisted in sessionStorage for subsequent page views) In addition, any variables set through Google Tag Manager (`window.dataLayer`) will be added as attributes as well. You can still specify your own custom attributes in the GrowthBook constructor, and they will be merged with the auto-attributes. #### Options By default the plugin generates a random UUID for the user and stores it in a cookie. You can customize this behavior with the following options. - `uuidKey` - The attribute key to use for the uuid (default is `id`) - `uuid` - Pass in your own user id instead of having the plugin generate a random one. - `uuidCookieName` - The name of the cookie to use for the generated uuid (default is `gb_uuid`) - `uuidCookieDomain` - Scope the uuid cookie to a parent domain (e.g. `".example.com"`) so the same anonymous id is shared across subdomains. Recommended if you run experiments that redirect users between subdomains, otherwise the visitor gets a new id (and can be re-bucketed) on the destination subdomain. - `uuidAutoPersist` - If `false`, the generated id will NOT be stored in a cookie automatically. Default depends on the `uuid` option. If `uuid` is set, the default is `false`. Otherwise, the default is `true`. If you have a cookie consent banner, you can set `uuidAutoPersist` to `false` and then later manually trigger the cookie to be saved by calling: ```js document.dispatchEvent(new CustomEvent("growthbookpersist")); ``` ## Loading Features In order for the GrowthBook SDK to work, it needs to have feature definitions from the GrowthBook API. There are 2 ways to get this data into the SDK. ### Built-in Fetching and Caching If you pass an `apiHost` and `clientKey` into the GrowthBook constructor, it will handle the network requests, caching, retry logic, etc. for you automatically. If your feature payload is encrypted, you can also pass in a `decryptionKey`. ```ts const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", // Only required if you have feature encryption enabled in GrowthBook decryptionKey: "key_abc123", }); await gb.init({ // If the network request takes longer than this (in milliseconds), continue // Default: `0` (no timeout) timeout: 2000, }) ``` Until features are loaded, all features will evaluate to `null`. If you're ok with a potential flicker in your application (features going from `null` to their real value), you can call `init` without awaiting the result. If you want to refresh the features at any time (e.g. when a navigation event occurs), you can call `gb.refreshFeatures()`. #### Error Handling In the case of network issues, the `init` call will not throw an error. Instead, it will stay in the default state where every feature evaluates to `null`. You can still get access to the error if needed: ```ts const res = await gb.init({ timeout: 1000 }); console.log(res); ``` The return value has 3 properties: - **status** - `true` if the GrowthBook instance was populated with features/experiments. Otherwise `false` - **source** - Where this result came from. One of the following values: `network`, `cache`, `init`, `error`, or `timeout` - **error** - If status is `false`, this will contain an `Error` object with more details about the error ### Custom Integration If you prefer to handle the network and caching logic yourself, you can pass in a full JSON "payload" directly into the SDK. For example, you might store features in Postgres and send it down to your front-end as part of your app's initial bootstrap API call. ```ts await gb.init({ payload: { features: { "feature-1": {...}, "feature-2": {...}, "another-feature": {...}, } } }) ``` The data structure for "payload" is exactly the same as what is returned by the GrowthBook SDK endpoints and webhooks. You can update the payload at any time by calling `setPayload(newPayloadJSON)` and there are also `getPayload()` and `getDecryptedPayload()` methods, which are useful in hybrid apps where you want to hydrate the client with data from the server. Note: you don't need to specify `clientKey` or `apiHost` on your GrowthBook instance unless you want to enable streaming (see below) or call `refreshFeatures()` later. #### Synchronous Init There is a alternate synchronous version of init named `initSync`, which can be especially useful in SSR to prevent hydration mismatches. There are some restrictions/differences: - You MUST pass in `payload` - The `payload` MUST NOT have encrypted features or experiments - If using sticky bucketing, you should use an instance of `StickyBucketServiceSync`, such as `BrowserCookieStickyBucketService`. - The return value is the GrowthBook instance to enable easy method chaining ## Waiting for Features to Load There is a helper component `` that lets you render a fallback component until features are done loading. This works for both built-in fetching and custom integrations. ```jsx }> ``` - `timeout` is the max time you want to wait for features to load (in ms). The default is `0` (no timeout). - `fallback` is the component you want to display before features are loaded. The default is `null`. If you want more control, you can use the `useGrowthBook()` hook and the `ready` flag: ```ts const gb = useGrowthBook(); if (gb.ready) { // Do something } ``` ## Streaming Updates The GrowthBook SDK supports streaming with Server-Sent Events (SSE). When enabled, changes to features within GrowthBook will be streamed to the SDK in realtime as they are published. This is only supported on GrowthBook Cloud or if running a GrowthBook Proxy Server. ### Streaming in Browser Environments SSE is supported on all major browsers, so enabling streaming is as easy as passing `streaming: true` into your `init` call: ```ts gb.init({ streaming: true, // Other settings... }) ``` You may also differentiate your streaming host URL from your API host by setting the `streamingHost` property in the GrowthBook constructor (ex: Remote Evaluation is done on a CDN edge worker while Streaming is done through a GrowthBook Proxy server). ### Streaming in ReactNative You need to install a polyfill for SSE to use streaming in a ReactNative application: ```bash npm2yarn npm install --save eventsource ``` The, tell GrowthBook to use this polyfill: ```js const { setPolyfills } = require("@growthbook/growthbook"); // Configure GrowthBook to use the eventsource library setPolyfills({ EventSource: require("eventsource"), }); ``` And finally, you can simply pass `streaming: true` into your init calls: ```ts gb.init({ streaming: true, // Other options... }) ``` ## Remote Evaluation :::tip See the [Remote Evaluation overview](/self-host/remote-evaluation) for more information about what Remote Evaluation is, how it works, and deployment options. ::: When used in a front-end context, the React SDK may be run in Remote Evaluation mode. This mode brings the security benefits of a backend SDK to the front end by evaluating feature flags exclusively on a private server. Using Remote Evaluation ensures that any sensitive information within targeting rules or unused feature variations are never seen by the client. Note that Remote Evaluation should not be used in a backend context (Hybrid SSR/CSR is also not supported). You must enable Remote Evaluation in your SDK Connection settings. Cloud customers are also required to self-host a GrowthBook Proxy Server, edge worker, or custom remote evaluation backend. To use Remote Evaluation, add the `remoteEval: true` property to your SDK instance. A new evaluation API call will be made any time a user attribute or other dependency changes. You may optionally limit these API calls to specific attribute changes by setting the `cacheKeyAttributes` property (an array of attribute names that, when changed, trigger a new evaluation call). ```ts const gb = new GrowthBook({ apiHost: "https://gb-proxy.mydomain.io/", clientKey: "sdk-abc123", // Enable remote evaluation remoteEval: true, // Optional: only trigger a new evaluation call when the `id` and `email` attribute changes cacheKeyAttributes: ["id", "email"], }); ``` :::note Sticky Bucketing with Remote Evaluation If you would like to implement Sticky Bucketing while using Remote Evaluation, you must configure your remote evaluation backend to support Sticky Bucketing. In the case of the GrowthBook Proxy Server, this means implementing a Redis database for sticky bucketing use. You will not need to provide a StickyBucketService instance to the client side SDK. ::: ## Caching The JavaScript SDK has 2 caching layers: 1. In-memory cache (available on all platforms) 2. Persistent localStorage cache (only available in browsers by default) There are a number of cache settings you can configure within GrowthBook. This must be done BEFORE creating a GrowthBook instance. Below are all of the default values. You can call `configureCache` with a subset of these fields and the rest will keep their default values. ```ts import { configureCache } from "@growthbook/growthbook"; configureCache({ // The localStorage key the cache will be stored under cacheKey: "gbFeaturesCache", // Consider features stale after this much time (60 seconds default) staleTTL: 1000 * 60, // Cached features older than this will be ignored (24 hours default) maxAge: 1000 * 60 * 60 * 24, // For Remote Eval only - limit the number of cache entries (~1 entry per user) maxEntries: 10, // When `false`, we add a `visibilitychange` listener to disable SSE when the page is idle disableIdleStreams: false, // Consider a page "idle" when it is hidden for this long (default 20 seconds) idleStreamInterval: 20000, // Set to `true` to completely disable both in-memory and persistent caching disableCache: false, }) ``` ### Polyfilling localStorage Outside of a browser environment, you can still use persistent caching. You just need to provide an implementation of the localStorage interface. Here's an example of using AsyncStorage within ReactNative: ```js import AsyncStorage from '@react-native-async-storage/async-storage'; import { setPolyfills } from "@growthbook/growthbook"; setPolyfills({ localStorage: { getItem: async (key) => { const value = AsyncStorage.getItem(key); return value != null ? JSON.parse(value) : null; }, setItem: async (key, value) => { await AsyncStorage.setItem(key, JSON.stringify(value)); } } }); ``` This must be done BEFORE you call either `prefetchPayload` or create the first GrowthBook instance. ## Experimentation (A/B Testing) In order to run A/B tests, you need to set up a tracking callback function. This is called every time a user is put into an experiment and can be used to track the exposure event in your analytics system (Segment, Mixpanel, GA, etc.). ```js const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", trackingCallback: (experiment, result, user) => { // Example using Segment analytics.track("Experiment Viewed", { experimentId: experiment.key, variationId: result.key, attributes: user?.attributes, }); }, }); ``` Starting in version 1.7.0, the callback receives a third `user` argument: a `TrackingUserContext` containing the `attributes` and optional `url` used when the experiment was evaluated. We recommend recording attributes from this argument so exposure events consistently reflect the attributes GrowthBook used for targeting and assignment. This same tracking callback is used for both feature flag experiments and Visual Editor experiments. ### Feature Flag Experiments There is nothing special you have to do for feature flag experiments. Just evaluate the feature flag like you would normally do. If the user is put into an experiment as part of the feature flag, it will call the `trackingCallback` automatically in the background. ```js // If this has an active experiment and the user is included, // it will call trackingCallback automatically useFeatureIsOn("new-signup-form") ``` If the experiment came from a feature rule, `result.featureId` in the trackingCallback will contain the feature id, which may be useful for tracking/logging purposes. ### Visual Editor Experiments Experiments created through the GrowthBook Visual Editor will run automatically as soon as their targeting conditions are met. **Note**: Visual Editor experiments are only supported in a web browser environment. They will not run in React Native or during Server Side Rendering (SSR). If you are using this SDK in a Single Page App (SPA), you will need to let the GrowthBook instance know when the URL changes so the active experiments can update accordingly. For example, in Next.js, you could do this: ```js function updateGrowthBookURL() { gb.setURL(window.location.href); } export default function MyApp() { // Subscribe to route change events and update GrowthBook const router = useRouter(); useEffect(() => { router.events.on("routeChangeComplete", updateGrowthBookURL); return () => router.events.off("routeChangeComplete", updateGrowthBookURL); }, []); // ... } ``` ### URL Redirect Experiments Similarly to Visual Editor experiments, URL redirect tests will run automatically if targeting conditions are met. If you are using this SDK in a Single Page App (SPA), you'll want to pass in a custom navigation function into the SDK (as default navigation for URL Redirects uses `window.location.replace(url)`) and set the `navigateDelay` to 0. ```js // Example in Next.js import router from "next/router"; const gb = new GrowthBook({ navigate: (url) => router.replace(url), navigateDelay: 0, // ... other settings }); ``` For SPA's you will also need to let the GrowthBook instance know when the URL changes so the active experiments can update accordingly. ```js // Call this every time a navigation event happens in your SPA function onRouteChange() { gb.setURL(window.location.href); } ``` ### Sticky Bucketing Sticky bucketing ensures that users see the same experiment variant, even when user session, user login status, or experiment parameters change. See the [Sticky Bucketing docs](/app/sticky-bucketing) for more information. If your organization and experiment supports sticky bucketing, you must implement an instance of the `StickyBucketService` to use Sticky Bucketing. The JS SDK exports several implementations of this service for common use cases, or you may build your own: - `LocalStorageStickyBucketService` — For simple bucket persistence using the browser's LocalStorage (can be polyfilled for other environments). - `BrowserCookieStickyBucketService` — For simple bucket persistence using browser cookies, which are transportable to the back end. Assumes `js-cookie` is implemented (can be polyfilled). Cookie attributes can also be configured. The default cookie expiry is 180 days; override by passing `expires: {days}` into the constructor's `cookieAttributes`. - Build your own — Implement the abstract `StickyBucketService` class and connect to your own data store, or custom wrap multiple service implementations (ex: read/write to both cookies and Redis). Implementing most StickyBucketService implementations is straightforward and works with minimal setup. For instance, to use the `BrowserCookieStickyBucketService`: ```js import { BrowserCookieStickyBucketService } from "@growthbook/growthbook"; import Cookies from 'js-cookie'; const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", stickyBucketService: new BrowserCookieStickyBucketService({ jsCookie: Cookies, }), // ... }); ``` ## Next.js If you are using Next.js, checkout our example apps for [Next with App Router](https://github.com/growthbook/examples/tree/main/next-js) and [Next with Pages Router](https://github.com/growthbook/examples/tree/main/next-js-pages). The examples above show how to use GrowthBook with a number of different rendering strategies and app setups and the App Router example has been updated to support the latest features in Next 14. ## Server Side Rendering (SSR) This SDK fully supports server side rendering with React. ### React Server Components If your framework supports the new React Server Components (RSC), welcome to the future! GrowthBook works great with modern React. First, if you are running experiments with GrowthBook, you will need to fire analytics tracking calls to record which variation a user is assigned. Analytics tools are often only supported client-side, so you can create a small Client Component first: ```tsx "use client"; import { TrackingData } from "@growthbook/growthbook-react"; // Helper component to track experiment views from server components export default function GrowthBookTracking({ data }: { data: TrackingData[] }) { useEffect(() => { data.forEach(({ experiment, result }) => { console.log("Viewed Experiment", { experimentId: experiment.key, variationId: result.key }); }); }, [data]) return null; } ``` The React SDK relies on client-side Context, so for Server Components, you need to import our Javascript SDK `@growthbook/growthbook` instead. ```tsx import { GrowthBook } from "@growthbook/growthbook"; import GrowthBookTracking from "./GrowthBookTracking"; export default async function MyServerPage() { // Create and initialize a GrowthBook instance const gb = new GrowthBook({ apiHost: process.env.NEXT_PUBLIC_GROWTHBOOK_API_HOST, clientKey: process.env.NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY, decryptionKey: process.env.NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY, }); await gb.init({ timeout: 1000 }); // Set targeting attributes for the user/page await gb.setAttributes({ // TODO: get this from cookies, headers, etc. id: cookies().get("my_uuid")?.value || "", }); // Evaluate any feature flags const showBanner = gb.isOn("showBanner"); const title = gb.getFeatureValue("title", "My Site"); // If the above features ran any experiments, get the tracking call data // This is passed into the client component below const trackingData = gb.getDeferredTrackingCalls(); // Cleanup gb.destroy(); return ( {title} {showBanner && ( There's a Sale! )} ); } ``` #### Hydrating Client Components From the Server The best part about React Server Components, is that you can easily share feature definitions with your Client Components. By doing this, you avoid any network requests from the browser and any flickering that goes along with that. Let's first create a GrowthBookWrapper wrapper that takes a `payload` prop and uses it to initialize a GrowthBook instance: ```tsx "use client"; import { GrowthBook, GrowthBookProvider, GrowthBookPayload } from "@growthbook/growthbook-react"; import { PropsWithChildren, useMemo } from "react"; import Cookies from "js-cookie"; export default function GrowthBookWrapper({ payload, children, }: PropsWithChildren) { // Create a singleton GrowthBook instance for this page const gb = useMemo( () => new GrowthBook({ apiHost: process.env.NEXT_PUBLIC_GROWTHBOOK_API_HOST, clientKey: process.env.NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY, decryptionKey: process.env.NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY, trackingCallback: (experiment, result, user) => { console.log("Viewed Experiment", { experimentId: experiment.key, variationId: result.key, attributes: user?.attributes, }); }, // Targeting attributes attributes: { id: Cookies.get("my_uuid"), }, }).initSync({ payload, // Optional, enable streaming updates streaming: true, }), [payload] ); return {children}; } ``` Now we'll make a really simple client component using the GrowthBook React SDK: ```tsx "use client"; import { useFeatureIsOn } from "@growthbook/growthbook-react"; export default function OtherComponent() { const clientFeature = useFeatureIsOn("client-feature"); return ( Client feature: {clientFeature ? "ON" : "OFF"} ) } ``` Now, we can render these from our server component: ```tsx import { GrowthBook } from "@growthbook/growthbook"; import GrowthBookWrapper from "./GrowthBookWrapper"; import OtherComponent from "./OtherComponent"; import GrowthBookTracking from "./GrowthBookTracking"; export default async function MyServerPage() { // Create and initialize a GrowthBook instance const gb = new GrowthBook({ apiHost: process.env.NEXT_PUBLIC_GROWTHBOOK_API_HOST, clientKey: process.env.NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY, decryptionKey: process.env.NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY, }); await gb.init({ timeout: 1000 }); // Set targeting attributes for the user gb.setAttributes({ id: cookies().get("my_uuid")?.value || "", }); // Evaluate any feature flags const serverFeature = gb.isOn("server-feature"); // If the above features ran any experiments, get the tracking call data // This is passed into the client component below const trackingData = gb.getDeferredTrackingCalls(); // Get the payload to hydrate the client-side GrowthBook instance // We need the decrypted payload so the initial client-render can be synchronous const payload = gb.getDecryptedPayload(); // Cleanup your GrowthBook instance gb.destroy(); return ( Server feature: {serverFeature ? "ON" : "OFF"} ); } ``` ### Traditional SSR Before React Server Components, each framework implemented their own way to do data fetching and SSR. This example uses Next.js `getServerSideProps` method, but other frameworks should be similar. With this approach, feature flags are evaluated once when the page is rendered. If a feature flag changes, the user would need to refresh the page to see it. ```js export const getServerSideProps = async (context) => { // Create and initialize a GrowthBook instance const gb = new GrowthBook({ apiHost: process.env.GROWTHBOOK_API_HOST, clientKey: process.env.GROWTHBOOK_CLIENT_KEY, decryptionKey: process.env.GROWTHBOOK_DECRYPTION_KEY, }); await gb.init({ timeout: 1000 }); // Set targeting attributes for the user await gb.setAttributes({ id: context?.cookies?.my_uuid || "", }); // Evaluate any feature flags const showBanner = gb.isOn("show-banner"); const title = gb.getFeatureValue("title", "My Site"); // If the above features ran any experiments, get the tracking call data // This is passed into the client component below const trackingData = gb.getDeferredTrackingCalls(); // Cleanup gb.destroy(); // Pass the result into your component return { props: { showBanner, title, trackingData } } } export default function MyPage({ title, showBanner, trackingData }) { useEffect(() => { trackingData?.forEach(({experiment, result}) => { // TODO: Track in your analytics tool console.log("Viewed Experiment", { experimentId: experiment.key, variationId: result.key }); }); }, [trackingData]) return ( {title} {showBanner && ( There's a Sale! )} ) } ``` #### Hybrid (SSR + Client-side) Instead of passing the result of individual feature flags to your component, you can also pass the entire payload. By doing this, you get the benefits of client-side rendering (interactivity, realtime feature flag updates) plus the benefits of SSR (no flickering, improved SEO). ```tsx export const getServerSideProps = async (context) => { // Create and initialize a GrowthBook instance const gb = new GrowthBook({ apiHost: process.env.NEXT_PUBLIC_GROWTHBOOK_API_HOST, clientKey: process.env.NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY, decryptionKey: process.env.NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY, }); await gb.init({ timeout: 1000 }); // Get the payload to hydrate the client-side GrowthBook instance // We need the decrypted payload so the initial client-render can be synchronous const payload = gb.getDecryptedPayload(); // Cleanup gb.destroy(); // Pass the result into your component return { props: { payload } } } export default function MyPage({ payload }) { // Create a singleton GrowthBook instance for this page const gb = useMemo( () => new GrowthBook({ apiHost: process.env.NEXT_PUBLIC_GROWTHBOOK_API_HOST, clientKey: process.env.NEXT_PUBLIC_GROWTHBOOK_CLIENT_KEY, decryptionKey: process.env.NEXT_PUBLIC_GROWTHBOOK_DECRYPTION_KEY, trackingCallback: (experiment, result, user) => { console.log("Viewed Experiment", { experimentId: experiment.key, variationId: result.key, attributes: user?.attributes, }); }, attributes: { id: Cookies.get("my_uuid"), }, }).initSync({ payload, // Optional, enable streaming updates streaming: true, }), [payload] ); return ; } ``` Then, within `MyComponent`, you can use any of the normal client-side hooks or helper components - `useFeatureIsOn`, `useFeatureValue`, etc. ## API Reference There are a number of configuration options and settings that control how GrowthBook behaves. ### Attributes You can specify attributes about the current user and request. These are used for two things: 1. Feature targeting (e.g. paid users get one value, free users get another) 2. Assigning persistent variations in A/B tests (e.g. user id "123" always gets variation B) The following are some commonly used attributes, but use whatever makes sense for your application. ```ts new GrowthBook({ attributes: { id: "123", loggedIn: true, deviceId: "abc123def456", company: "acme", paid: false, url: "/pricing", browser: "chrome", mobile: false, country: "US", }, }); ``` #### Updating Attributes If attributes change, you can call `setAttributes()` to update. This will completely overwrite any existing attributes. To do a partial update, use the following pattern: ```js gb.setAttributes({ // Only update the `url` attribute, keep the rest the same ...gb.getAttributes(), url: "/new-page" }) ``` #### Secure Attributes When _secure attribute hashing_ is enabled, all targeting conditions in the SDK payload referencing attributes with datatype `secureString` or `secureString[]` will be anonymized via SHA-256 hashing. This allows you to safely target users based on sensitive attributes. You must enable this feature in your SDK Connection for it to take effect. If your SDK Connection has secure attribute hashing enabled, you will need to manually hash any `secureString` or `secureString[]` attributes that you pass into the GrowthBook SDK. To hash an attribute, use a cryptographic library with SHA-256 support, and compute the SHA-256 hashed value of your attribute _plus_ your organization's secure attribute salt. The example below is using CryptoJS (https://www.npmjs.com/package/crypto-js), which provides a synchronous API. ```js import sha256 from 'crypto-js/sha256'; const salt = "f09jq3fij"; // Your secure attribute salt (set in Organization Settings) // hashing a secureString attribute const userEmail = sha256(salt + user.email).toString(); // hashing a secureString[] attribute const userTags = user.tags.map(tag => sha256(salt + tag).toString()); gb.setAttributes({ id: user.id, loggedIn: true, email: userEmail, tags: userTags, }); await gb.init(); ``` If you prefer to use a browser native implementation, you will need to await hashing before calling `setAttributes` since SubtleCrypto methods are async. Here's an example implementation: ```js // Native browser implementation async function sha256(str) { const buffer = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(str)); const hashArray = Array.from(new Uint8Array(buffer)); return hashArray.map(byte => byte.toString(16).padStart(2, "0")).join(""); } // Hashing a single string const userEmail = await sha256(salt + user.email); // Hashing an array of strings const userTags = await Promise.all( user.tags.map(tag => sha256(salt + tag)) ); ``` ### Feature Usage Callback GrowthBook can fire a callback whenever a feature is evaluated for a user. This can be useful to update 3rd party tools like NewRelic or DataDog. ```ts new GrowthBook({ onFeatureUsage: (featureKey, result) => { console.log("feature", featureKey, "has value", result.value); }, }); ``` Note: If you evaluate the same feature multiple times (and the value doesn't change), the callback will only be fired the first time. ### Dev Mode There is a [GrowthBook Chrome DevTools Extension](https://chrome.google.com/webstore/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) that can help you debug and test your feature flags in development. In order for this to work, you must explicitly enable dev mode when creating your GrowthBook instance: ```js const gb = new GrowthBook({ enableDevMode: true, }); ``` To avoid exposing all of your internal feature flags and experiments to users, we recommend setting this to `false` in production in most cases. ### Inline Experiments Depending on how you configure feature flags, they may run A/B tests behind the scenes to determine which value gets assigned to the user. Sometimes though, you want to run an inline experiment without going through a feature flag first. For this, you can use either the `useExperiment` hook or the Higher Order Component `withRunExperiment`: View the [Javascript SDK Docs](/lib/js) for all of the options available for inline experiments #### useExperiment hook ```tsx import { useExperiment } from "@growthbook/growthbook-react"; export default function OtherComponent() { const { value } = useExperiment({ key: "new-headline", variations: ["Hello", "Hi", "Good Day"] }); return {value}; } ``` #### withRunExperiment (class components) **Note:** This library uses hooks internally, so still requires React 16.8 or above. ```tsx import { withRunExperiment } from "@growthbook/growthbook-react"; class OtherComponent extends React.Component { render() { // The `runExperiment` prop is identical to the `useExperiment` hook const { value } = this.props.runExperiment({ key: "headline-test", variations: ["Hello World", "Hola Mundo"] }); return {value}; } } // Wrap your component in `withRunExperiment` export default withRunExperiment(OtherComponent); ``` ## TypeScript support Some hooks are available in type-safe versions. These require you to pass in your generated types as the generic argument. See the [GrowthBook CLI](/tools/cli) documentation for more information on generating type definitions and [JavaScript → TypeScript → Strict Typing](/lib/js#strict-typing) for how to use them. ### useGrowthBook<T>() A type-safe version of the `useGrowthBook()` hook is available. Everywhere you use `useGrowthBook()`, pass the generated features as the generic argument: ```ts const growthbook = useGrowthBook() ``` In that case, the hook will return `GrowthBook | undefined`. You can reduce this boilerplate by creating your own hook, e.g.: ```ts // ./src/utils/growthbook.ts import { useGrowthBook as _useGrowthBook } from "@growthbook/growthbook-react"; export const useGrowthBook = (): GrowthBook | undefined => _useGrowthBook(); ``` You can now reference the hook you created instead of the one from the official package: ```ts import { useGrowthBook } from "@/src/utils/growthbook" const growthbook = useGrowthBook(); growthbook.getFeatureValue(knownKey, defaultValueOfValidType) ``` ### useFeatureIsOn<T>() The React SDK also provides access to a type-safe `useFeatureIsOn()` hook. ```ts const isDarkModeOn = useFeatureIsOn("dark_mode"); ``` This will only allow you to pass known keys to the hook. You can reduce the boilerplate for this hook by creating your own and using that instead: ```ts // ./src/utils/growthbook.ts import { useFeatureIsOn as _useFeatureIsOn } from "@growthbook/growthbook-react"; export const useFeatureIsOn = (id: keyof AppFeatures & string): boolean => _useFeatureIsOn(id); ``` And then reference the hook you created instead of the one from the official package: ```ts import { useFeatureIsOn } from "@/src/utils/growthbook" const isDarkModeOn = useFeatureIsOn("dark_mode"); ``` ## Examples - [Next.js ](https://github.com/growthbook/examples/tree/main/next-js) - [React Native ](https://github.com/growthbook/examples/tree/main/react-native-cli) - [Typescript example app with strict typing ](https://github.com/growthbook/examples/tree/main/vanilla-typescript). ## Supported Features --- ## Roku SDK import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' # Roku SDK Official GrowthBook SDK for Roku/js applications. Add feature flags and A/B testing to your Roku channels with a simple, lightweight SDK. ## Platform Requirements The GrowthBook SDK supports all modern Roku devices and OS versions: - **Minimum OS Version:** Roku OS 9.0+ - **Recommended:** Roku OS 9.2+ (for AES encryption support) - **Tested on:** Roku OS 9.0 - 12.x **Device Compatibility:** - ✅ Roku Ultra (all versions) - ✅ Roku Streaming Stick (all versions) - ✅ Roku Express - ✅ Roku Premiere - ✅ Roku TV - ✅ Legacy devices (Roku 2, Roku 3) The SDK is written in pure JS and requires: - **js 1.0+** - Core language features - **SceneGraph Support** - For modern Roku channel development - **roUrlTransfer** - For API communication (standard in all Roku OS versions) - **roEVPCipher** (Optional) - For encrypted payloads (Roku OS 9.2+) **No External Dependencies:** The SDK is completely self-contained with zero external dependencies, making it easy to integrate into any Roku channel. - **Core SDK File:** ~50 KB (single `GrowthBook.brs` file) - **Runtime Memory:** <500 KB total footprint - **Feature Evaluation:** <1ms per check - **Network Calls:** One-time feature load on init (cached for instance lifetime) **Performance Tested On:** - Roku Ultra 2023 (high-end reference) - Roku Express 2022 (low-end reference) - Roku Streaming Stick 4K ## Installation ### Manual Installation (Recommended) 1. Download `GrowthBook.brs` from the [GitHub repository](https://github.com/growthbook/growthbook-roku/tree/main/source). 2. Copy it to your channel's `source/` directory: ```text your-roku-channel/ ├── source/ │ ├── main.brs │ └── GrowthBook.brs ← Add this file └── manifest ``` ### Installation with ropm If you are using [ropm](https://github.com/rokucommunity/ropm) for dependency management: ```bash ropm install growthbook-roku ``` ## Quick Usage ### Step 1: Initialize the SDK Initialize GrowthBook once when your channel starts. Use a **singleton pattern** to reuse the instance throughout your app. ```js sub Main() ' Create global field for GrowthBook instance m.global.addFields({ gb: invalid }) ' Initialize GrowthBook once m.global.gb = GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: { id: GetDeviceId(), deviceType: "roku", appVersion: "1.0.0" } }) ' Load features if m.global.gb.init() print "GrowthBook ready!" else print "GrowthBook failed to initialize" end if ' Start your app ShowHomeScreen() end sub function GetDeviceId() as string deviceInfo = CreateObject("roDeviceInfo") return deviceInfo.GetChannelClientId() end function ``` ### Step 2: Use Feature Flags Once initialized, access the GrowthBook instance anywhere in your channel: ```js ' Boolean feature flag if m.global.gb.isOn("new-player-ui") then ShowNewPlayer() else ShowLegacyPlayer() end if ' Feature value with fallback buttonColor = m.global.gb.getFeatureValue("cta-color", "#0000FF") maxVideos = m.global.gb.getFeatureValue("videos-per-page", 12) ' JSON configuration playerConfig = m.global.gb.getFeatureValue("player-settings", { autoplay: false, quality: "HD" }) ``` ## Loading Features The GrowthBook SDK provides multiple strategies for loading feature flags, allowing you to choose between automated feature loading at the init or using it in offline mode. ### Automated Loading At the time of initialization, the SDK automatically fetches features from the GrowthBook API when you provide `apiHost` and `clientKey`: ```js gb = GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk_abc123", attributes: { id: GetDeviceId() } }) ' Automatically fetches features from API if gb.init() then print "Features loaded successfully" else print "Failed to load features" end if ``` **How It Works:** 1. When `init()` is called, the SDK makes an HTTP request to `{apiHost}/api/features/{clientKey}` 2. Features are cached in memory for the lifetime of the instance 3. All subsequent feature evaluations use the cached data (no additional network calls) **Feature Refresh Frequency:** - Features are loaded **once** when `init()` is called - **No automatic background refresh** is supported for now. - **Roku Limitation:** Unlike web SDKs, Roku does not support Server-Sent Events (SSE) for real-time streaming updates - To get updated features, you must reinitialize the GrowthBook instance (typically on app restart) - For real-time updates, implement a manual refresh mechanism: ```js ' Manual refresh example (requires creating new instance) sub RefreshFeatures() oldAttributes = m.global.gb.attributes ' Create new instance with same config m.global.gb = GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk_abc123", attributes: oldAttributes }) ' Reload from API if m.global.gb.init() then print "Features refreshed successfully" end if end sub ``` ### Offline Mode For scenarios where network access is unavailable or you want to embed features directly in your channel, use offline mode. ```js ' Pre-define features in your code defaultFeatures = { "enable-4k-streaming": { defaultValue: true }, "videos-per-page": { defaultValue: 12 }, "player-settings": { defaultValue: { autoplay: false, quality: "HD", showCaptions: true } }, "button-color": { rules: [{ variations: ["blue", "red", "green"], weights: [0.5, 0.3, 0.2] }] } } ' Initialize with embedded features gb = GrowthBook({ features: defaultFeatures, attributes: { id: GetDeviceId() } ' Note: Do NOT provide clientKey or apiHost in offline mode }) ' init() returns true immediately - ZERO network calls made gb.init() ``` **When to Use Offline Mode:** - ✅ Testing and development without GrowthBook account - ✅ Regions with unreliable or no network connectivity - ✅ Regulatory requirements preventing external API calls - ✅ Feature flags that rarely change and can be bundled with app - ✅ Fallback strategy for network failures - ✅ Kiosk or offline-first applications **Limitations:** - ❌ Features must be updated via channel deployment (sideload or store update) - ❌ Cannot change feature values remotely without app update - ❌ No real-time experimentation updates ## Configuration Options ### Required Options At minimum, provide either `clientKey` OR `features`: ```js ' Option 1: API loading gb = GrowthBook({ clientKey: "sdk_abc123", attributes: { id: GetDeviceId() } }) ' Option 2: Offline mode gb = GrowthBook({ features: myFeatures, attributes: { id: GetDeviceId() } }) ``` ### All Configuration Options ```js gb = GrowthBook({ ' API Configuration apiHost: "https://cdn.growthbook.io", ' API endpoint clientKey: "sdk_abc123", ' SDK client key ' User Context attributes: { ' User attributes for targeting id: GetDeviceId(), appVersion: "1.0.0", country: "US", subscription: "premium" }, ' Callbacks trackingCallback: sub(experiment, result) ' Experiment exposure tracking SendAnalyticsEvent(experiment, result) end sub, onFeatureUsage: sub(featureKey, result) ' All feature evaluations LogFeatureUsage(featureKey, result) end sub, ' Advanced enableDevMode: true, ' Debug logging features: defaultFeatures, ' Offline/fallback features savedGroups: { ' Saved user groups "beta_testers": ["user1", "user2"] }, forcedVariations: { ' Force specific variations "feature-key": 1 } }) ``` ## Instance Management (Singleton Pattern) **Recommended**: Create one instance and reuse it globally. ```js ' In your main entry point sub Main() ' Create global field once m.global.addFields({ gb: invalid }) ' Initialize once m.global.gb = GrowthBook({ clientKey: "sdk_YOUR_KEY", attributes: { id: GetDeviceId() } }) m.global.gb.init() ' Now use throughout app ShowApp() end sub ' In any component or function function CheckFeature() ' Access the same instance if m.global.gb.isOn("my-feature") then DoSomething() end if end function ``` Creating multiple GrowthBook instances causes problems: | Problem | Impact | | ------------------- | ------------------------------------------------------ | | Redundant API calls | Every `init()` call fetches features again (expensive) | | Memory usage | Each instance consumes ~150KB of memory | | Poor performance | Network latency on every feature evaluation | ### Updating Attributes Update user attributes without recreating the instance: ```js ' ✅ DO THIS: Update attributes m.global.gb.setAttributes({ id: newUserId, subscription: "premium", country: "US" }) ' ❌ DON'T DO THIS: Recreate instance m.global.gb = GrowthBook({ ... }) ' Wrong! m.global.gb.init() ' Wrong! ``` ## Feature Flags There are 2 main methods for evaluating features: `isOn` and `getFeatureValue`: ### Boolean Flags Simple on/off toggles: ```js if m.global.gb.isOn("dark-mode") then ApplyDarkTheme() end if if m.global.gb.isOn("enable-4k") then Enable4KStreaming() end if ``` ### Feature Values Get configuration values with type-safe fallbacks: ```js ' Strings buttonColor = m.global.gb.getFeatureValue("button-color", "#0000FF") apiEndpoint = m.global.gb.getFeatureValue("api-url", "https://api.example.com") ' Numbers maxRetries = m.global.gb.getFeatureValue("max-retries", 3) timeout = m.global.gb.getFeatureValue("timeout-ms", 5000) ' Booleans autoplay = m.global.gb.getFeatureValue("autoplay-enabled", false) ``` ### JSON Configuration Complex objects for advanced configuration: ```js playerSettings = m.global.gb.getFeatureValue("player-config", { autoplay: false, quality: "HD", subtitles: true, bufferSize: 5, maxBitrate: 10000 }) ' Apply settings m.player.autoplay = playerSettings.autoplay m.player.quality = playerSettings.quality m.player.subtitles = playerSettings.subtitles ``` ## Experimentation (A/B Testing) There is nothing special you have to do for feature flag experiments. Just evaluate the feature flag like you would normally do. If the user is put into an experiment as part of the feature flag, it will call the `trackingCallback` automatically in the background. ```js result = m.global.gb.getFeatureValue("pricing-experiment") print "Feature key: " + result.key print "Value: " + Str(result.value) print "Enabled: " + Str(result.on) print "Source: " + result.source ' "defaultValue", "force", "experiment" if result.source = "experiment" then print "Experiment: " + result.experimentId print "Variation: " + Str(result.variationId) print "Rule: " + result.ruleId end if ``` ### Tracking Callbacks GrowthBook provides two types of callbacks to monitor feature usage and experiment exposure: #### Experiment Tracking Callback The `trackingCallback` is fired when a user is placed into an experiment. Use this to send experiment exposure events to your analytics platform (Segment, Mixpanel, Google Analytics, etc.). ```js gb = GrowthBook({ clientKey: "sdk_abc123", attributes: { id: GetDeviceId() }, trackingCallback: sub(experiment, result) ' Send to your analytics platform SendAnalyticsEvent({ event: "experiment_viewed", experimentId: result.experimentId, variationId: result.variationId, value: result.value }) end sub }) ``` **Callback Parameters:** **`experiment` object:** - `key` (string) - Experiment identifier - `variations` (array) - List of possible variations - `weights` (array) - Traffic allocation weights - `hashVersion` (integer) - Hash algorithm version - `namespace` (array) - Namespace for traffic allocation **`result` object:** - `experimentId` (string) - Experiment key - `variationId` (integer) - Assigned variation index (0-based) - `value` (dynamic) - Actual variation value - `ruleId` (string) - ID of the rule that triggered - `source` (string) - Always `"experiment"` for tracking callback - `on` (boolean) - Whether feature is enabled - `key` (string) - Feature key **When It Fires:** - User enters an experiment (assigned to a variation) - Only fires ONCE per unique experiment + user combination (de-duplicated) **Does NOT fire for:** - Features with no experiments - Users excluded from experiments - Forced variations #### Feature Usage Callback The `onFeatureUsage` callback is fired on **every** feature evaluation, not just experiments. Use this for high-level usage tracking or debugging. ```js gb = GrowthBook({ clientKey: "sdk_abc123", attributes: { id: GetDeviceId() }, onFeatureUsage: sub(featureKey, result) print "Feature evaluated: " + featureKey print "Value: " + Str(result.value) print "Source: " + result.source ' Optional: Send to analytics for all feature usage if result.source = "experiment" then TrackExperimentUsage(featureKey, result) end if end sub }) ``` **Callback Parameters:** - `featureKey` (string) - The key of the feature being evaluated - `result` (object) - Complete evaluation result (same structure as `evalFeature()`) **When It Fires:** - Every call to `isOn()`, `getFeatureValue()`, or `evalFeature()` - Includes all sources: `"defaultValue"`, `"force"`, `"experiment"`, `"unknownFeature"` #### Comparison | | `trackingCallback` | `onFeatureUsage` | | ------------------ | --------------------------------- | ----------------------------- | | **Purpose** | Track experiment exposures | Track all feature evaluations | | **Frequency** | Once per experiment + user | Every feature check | | **Use Case** | Send to analytics for A/B testing | Debugging, usage monitoring | | **Fires For** | Experiments only | All features | | **De-duplication** | Yes (automatic) | No | ### Hashing and Consistent Assignment GrowthBook uses deterministic hashing to ensure users get consistent variation assignments. This is critical for accurate A/B testing. #### How Hashing Works When a user is evaluated for an experiment: 1. **Hash Input:** The SDK combines the user's `id` attribute with the experiment key 2. **Generate Hash:** Uses FNV-1a hashing algorithm to produce a number between 0 and 1 3. **Map to Bucket:** The hash maps to a specific variation based on traffic weights 4. **Return Variation:** User is assigned to that variation consistently ```js ' User with id "user_12345" evaluates feature "button_color" ' SDK calculates: hash("button_color", "user_12345") → 0.742 ' Traffic split: [0-0.5: blue, 0.5-0.8: red, 0.8-1.0: green] ' 0.742 falls in red bucket → user sees red ' EVERY TIME this user checks, they see red (consistent) ``` #### Hash Attribute By default, GrowthBook uses the `id` attribute for hashing. You can customize which attribute to use with the `hashAttribute` setting in your experiment rules. **Default Behavior:** ```js gb = GrowthBook({ attributes: { id: "user_12345", // Used for hashing by default email: "user@example.com", subscription: "premium" } }) ' Hashes using "id" attribute color = gb.getFeatureValue("button_color", "blue") ``` **Custom Hash Attribute:** In the GrowthBook dashboard, set `hashAttribute` in your experiment rule: ```javascript { "key": "button-color-experiment", "hashAttribute": "deviceId", // Use deviceId instead of id "variations": ["blue", "red", "green"], "weights": [0.5, 0.3, 0.2] } ``` ```js ' Now SDK uses "deviceId" for hashing gb = GrowthBook({ attributes: { id: "user_12345", deviceId: "roku_device_xyz", // This is used for hashing email: "user@example.com" } }) color = gb.getFeatureValue("button_color", "blue") ' Variation determined by hash("button-color-experiment", "roku_device_xyz") ``` #### When to Use Custom Hash Attributes Use custom hash attributes when: | Use Case | Hash Attribute | Reason | | ---------------------------- | -------------- | ---------------------------------------------- | | **Device-level experiments** | `deviceId` | Same variation across all users on that device | | **Household experiments** | `accountId` | Same variation for all family members | | **Company-level B2B** | `companyId` | Consistent experience for all employees | | **Anonymous users** | `sessionId` | Consistent during session only | | **Cross-platform sync** | `userId` | Same variation on web, mobile, Roku | **Example: Device-Level Experiment** ```js ' Use case: Test new UI for entire household (device) gb = GrowthBook({ attributes: { id: "anonymous_user", deviceId: GetDeviceId(), // Roku device ID householdId: "household_789" } }) ' In dashboard, set hashAttribute: "deviceId" ' Now all users on this Roku device see the same UI variant if gb.isOn("new-ui-test") then ShowNewUI() end if ``` #### Debugging Hash Assignments Enable dev mode to see hash calculations: ```js gb = GrowthBook({ clientKey: "sdk_key", attributes: { id: "user_12345" }, enableDevMode: true ' ← Enable debug logging }) ' Console output: ' [GrowthBook] Evaluating experiment: button-color-test ' [GrowthBook] Hash input: button-color-test|user_12345 ' [GrowthBook] Hash value: 0.742 ' [GrowthBook] Assigned to variation 1 (red) ``` ## Production Best Practices ### Error Handling Always check if GrowthBook initialized successfully: ```js gb = GrowthBook({ clientKey: "sdk_key", attributes: { id: GetDeviceId() } }) if not gb.init() then print "GrowthBook failed to initialize - using defaults" ' Continue with app using fallback values end if ' Safe feature access function SafeFeatureCheck(key as string, default as dynamic) as dynamic if m.global.gb invalid then return m.global.gb.getFeatureValue(key, default) end if return default end function ``` ### Performance Optimization Avoid calling feature checks in tight loops: ```js ' ❌ BAD: Feature check in loop for each video in videos maxQuality = gb.getFeatureValue("max-quality", "HD") video.quality = maxQuality end for ' ✅ GOOD: Check once, use many times maxQuality = gb.getFeatureValue("max-quality", "HD") for each video in videos video.quality = maxQuality end for ``` ## Troubleshooting ### 1) SDK Not Loading `GrowthBook()` returns `invalid` 1. Verify `GrowthBook.brs` is in `source/` directory 2. Check file name is exactly `GrowthBook.brs` (case-sensitive) 3. Ensure no syntax errors in the file 4. Try compiling channel to see errors ### 2) Features Not Loading `init()` returns `false` 1. Check `clientKey` is correct 2. Verify network connectivity 3. Test API endpoint: `https://cdn.growthbook.io/api/features/YOUR_KEY` 4. Enable dev mode: `enableDevMode: true` to see error logs 5. Provide fallback features for offline resilience ### 3) Version Targeting Not Working Version-based rules don't match 1. Use semantic versioning: `"2.1.0"` not `"2.1"` or `"v2.1.0"` 2. Verify `appVersion` attribute is set correctly 3. Test version operators in GrowthBook dashboard preview 4. Enable dev mode to see evaluation logs ### 4) Inconsistent Variations User sees different variations across sessions 1. Ensure `id` attribute is stable (use device ID, not random) 2. Don't use `Rnd()` or random values for `id` 3. Verify `id` is set before evaluating features 4. Check that you're not creating multiple GrowthBook instances ### 5) Experiments Show Wrong Traffic Split 50/50 split when expecting 70/30 1. Verify weights in GrowthBook dashboard match expectations 2. Ensure weights array length matches variations count 3. Test with multiple user IDs to verify distribution 4. Check experiment is published and active ## Limitations - ❌ No Server-Sent Events (SSE) streaming support (Roku limitation) - ❌ No Visual Editor experiments (SceneGraph only) - ❌ AES decryption requires Roku OS 9.2+ (`roEVPCipher` component) - ❌ Network requests are asynchronous only (no sync API) ## Supported Features --- ## Ruby SDK import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Ruby ## Requirements The Ruby SDK requires Ruby version 2.5.0 or higher. ## Installation Install the gem: ``` gem install growthbook ``` ## Quick start ```ruby require 'growthbook' # Fetch features from a GrowthBook instance # You should cache this in Redis or similar in production features_repository = Growthbook::FeatureRepository.new( endpoint: 'https://cdn.growthbook.io/api/features/MY_API_KEY', decryption_key: nil ) features = features_repository.fetch # Create a context for the current user/request gb = Growthbook::Context.new( features: features, # User attributes for targeting / variation assignment attributes: { id: '123', country: 'US' } ) # Use a boolean feature flag if gb.on? :my_feature_key puts 'My feature is on!' end # Get the value of a multivariate feature with a fallback btn_color = gb.feature_value(:signup_btn_color, 'pink') ``` ## Tracking ### Track experiment impressions When a feature's value is determined by an experiment (A/B test), you typically want to track that assignment event for later analysis. There are two ways to do this. First is by accessing all impressions at the end of a request: ```ruby gb.impressions.each do |key, result| puts "Assigned variation #{result.variation_id} in experiment #{key}" end ``` Second is by using a listener to get alerted in realtime as users are put into experiments: ```ruby class MyImpressionListener def on_experiment_viewed(experiment, result) puts "Assigned variation #{result.variation_id} in experiment #{experiment.key}" end end gb.listener = MyImpressionListener.new ``` ### Track feature usage GrowthBook can fire a callback whenever a feature is evaluated for a user. This can be useful to update 3rd party tools like NewRelic or DataDog. Provide a receiver that can receive `def on_feature_usage: (String _feature_key, FeatureResult _result) -> void`. There's a convenience class `FeatureUsageCallback` with a method you can override but you can provide your own. ```rb class MyFeatureUsageCallback ', attributes: { id: '456', country: 'CA' } ) ``` ## Code Examples - [Ruby on Rails example ](https://github.com/growthbook/examples/tree/main/acme_donuts_rails) ## Further Reading - [Generated class docs ](https://growthbook.github.io/growthbook-ruby/) ## Supported Features --- ## Rust SDK import ExternalLink from '@site/src/components/ExternalLink' import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Rust SDK The official GrowthBook SDK for Rust. This SDK provides a powerful, type-safe way to integrate feature flagging and A/B testing into your Rust applications with automatic feature refreshing, caching, and tracking callbacks. ## Requirements - **Rust 1.70.0 or higher** (as specified in `rust-toolchain`) - Async runtime: Tokio (recommended) or any `async-std` compatible runtime ## Installation Add this to your `Cargo.toml`: ```toml [dependencies] growthbook-rust = "0.0.4" tokio = { version = "1", features = ["full"] } serde_json = "1.0" ``` Or install via cargo: ```bash cargo add growthbook-rust ``` ## Quick Usage ### Step 1: Initialize the Client ```rust use growthbook_rust::client::GrowthBookClientBuilder; use std::time::Duration; #[tokio::main] async fn main() -> Result> { // Create a GrowthBook client with auto-refresh enabled let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .ttl(Duration::from_secs(60)) // Cache TTL .auto_refresh(true) // Enable background updates .refresh_interval(Duration::from_secs(30)) .build() .await?; Ok(()) } ``` ### Step 2: Evaluate Feature Flags ```rust // Simple boolean check if client.is_on("my-feature", None) { println!("Feature is enabled!"); } // Get typed feature value let result = client.feature_result("button-color", None); if let Ok(color) = result.value_as::() { println!("Button color: {}", color); } // With user attributes use growthbook_rust::model_public::{GrowthBookAttribute, GrowthBookAttributeValue}; use std::collections::HashMap; let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String("user-123".to_string()) )); if client.is_on("premium-feature", Some(attrs)) { println!("Premium feature enabled for this user!"); } ``` ## Loading Features and Experiments The Rust SDK provides multiple ways to load and refresh feature definitions from the GrowthBook API. ### Built-in Fetching and Auto-Refresh The recommended approach is to use the builder with auto-refresh enabled: ```rust use growthbook_rust::client::GrowthBookClientBuilder; use std::time::Duration; #[tokio::main] async fn main() -> Result> { let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) // Cache settings .ttl(Duration::from_secs(60)) // How long to cache features // Auto-refresh settings .auto_refresh(true) // Enable background sync .refresh_interval(Duration::from_secs(30)) // Refresh every 30 seconds .build() .await?; // Features are now loaded and will refresh automatically // in the background every 30 seconds Ok(()) } ``` #### How Auto-Refresh Works When `auto_refresh` is enabled: 1. Features are fetched immediately during `build()` 2. A background task spawns that periodically fetches updates 3. The cache is updated automatically without blocking your application 4. The refresh task runs until the client is dropped **Benefits:** - Always up-to-date features without manual intervention - Non-blocking updates in the background - Configurable refresh intervals - Automatic retry logic on failures ### Manual Refresh You can manually trigger a feature refresh at any time: ```rust // Force a feature refresh client.refresh().await; // Useful for scenarios like: // - User login/logout events // - Navigation changes // - Manual override in admin panels ``` ### Starting with Initial Features If you want to start with a specific set of features (e.g., from a file or cache) and then enable updates: ```rust use serde_json::json; let initial_features = json!({ "my-feature": { "defaultValue": true }, "button-color": { "defaultValue": "blue" } }); let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .features_json(initial_features)? // Start with these features .auto_refresh(true) // Still enable auto-refresh .refresh_interval(Duration::from_secs(30)) .build() .await?; // Client starts with initial features and updates them in the background ``` ### Disabling Auto-Refresh For use cases where you want full control (e.g., testing, edge workers, or custom update logic): ```rust let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .auto_refresh(false) // Disable background sync .build() .await?; // Manually control when to refresh client.refresh().await; ``` ### Refresh Callback Get notified when features are refreshed (useful for logging and debugging): ```rust let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .auto_refresh(true) .refresh_interval(Duration::from_secs(30)) .add_on_refresh(Box::new(|| { println!("✅ Features refreshed at {}", chrono::Utc::now()); // Update metrics, logs, or caches })) .build() .await?; ``` ### Configuration via Environment Variables The SDK supports configuration through environment variables: | Variable | Description | Default | | ------------------------ | --------------------- | ---------- | | `GB_HTTP_CLIENT_TIMEOUT` | HTTP request timeout | 10 seconds | | `GB_UPDATE_INTERVAL` | Auto-refresh interval | 60 seconds | | `GB_URL` | GrowthBook API URL | - | | `GB_SDK_KEY` | SDK client key | - | ```rust // Environment variables will be used if not explicitly set let client = GrowthBookClientBuilder::new() // api_url and client_key will be read from GB_URL and GB_SDK_KEY .build() .await?; ``` ## Encrypted Features For enhanced security, GrowthBook supports encrypted feature payloads. This prevents sensitive feature configurations and PII data from being exposed in transit or in logs. ### Setup 1. **Enable encryption** in your GrowthBook SDK Connection settings 2. **Copy the decryption key** shown in the GrowthBook dashboard 3. **Pass the key to the SDK** during initialization ```rust use growthbook_rust::client::GrowthBookClientBuilder; let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .decryption_key("your-decryption-key-here".to_string()) // Add this .build() .await?; // Features are automatically decrypted when loaded ``` ### How It Works - Feature payloads from the API are encrypted using AES-256 - The SDK automatically decrypts them using your decryption key - Decryption happens transparently - your code doesn't change - Invalid keys or corrupted data will cause initialization to fail ### Security Best Practices ```rust use std::env; // ✅ DO: Load from environment variables let decryption_key = env::var("GROWTHBOOK_DECRYPTION_KEY") .expect("GROWTHBOOK_DECRYPTION_KEY must be set"); let client = GrowthBookClientBuilder::new() .api_url(env::var("GROWTHBOOK_API_URL").unwrap()) .client_key(env::var("GROWTHBOOK_CLIENT_KEY").unwrap()) .decryption_key(decryption_key) .build() .await?; // ❌ DON'T: Hardcode keys in source code // let decryption_key = "key-123456789".to_string(); // NEVER DO THIS! ``` **Recommendations:** - Use environment variables or secret management systems (AWS Secrets Manager, HashiCorp Vault) - Rotate keys regularly - Use different keys for different environments (dev, staging, production) - Never commit keys to version control ### Error Handling ```rust let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .decryption_key("wrong-key".to_string()) .build() .await; match client { Ok(client) => { println!("Client initialized successfully"); }, Err(e) => { eprintln!("Failed to initialize client: {}", e); // Could be due to: // - Wrong decryption key // - Network issues // - Invalid client key // Fall back to safe defaults } } ``` ## Attributes Attributes are used for two main purposes: 1. **Feature targeting** - Show different values to different user segments 2. **Experiment bucketing** - Ensure consistent variation assignment ### Setting Global Attributes You can set default attributes that apply to all feature evaluations: ```rust use std::collections::HashMap; use growthbook_rust::model_public::GrowthBookAttributeValue; // Define global attributes during client creation let mut global_attrs = HashMap::new(); global_attrs.insert( "tenantId".to_string(), GrowthBookAttributeValue::String("acme-corp".to_string()) ); global_attrs.insert( "plan".to_string(), GrowthBookAttributeValue::String("enterprise".to_string()) ); let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .attributes(global_attrs) // Set global attributes .build() .await?; ``` ### Per-Evaluation Attributes You can override or supplement global attributes on a per-check basis: ```rust use growthbook_rust::model_public::{GrowthBookAttribute, GrowthBookAttributeValue}; // Create per-evaluation attributes let mut user_attrs = Vec::new(); user_attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String("user-456".to_string()) )); user_attrs.push(GrowthBookAttribute::new( "country".to_string(), GrowthBookAttributeValue::String("US".to_string()) )); user_attrs.push(GrowthBookAttribute::new( "isPremium".to_string(), GrowthBookAttributeValue::Bool(true) )); // These attributes are merged with global attributes if client.is_on("new-dashboard", Some(user_attrs)) { // Show new dashboard } ``` ### Attribute Types The SDK supports all JSON data types as attributes: ```rust use growthbook_rust::model_public::GrowthBookAttributeValue; use serde_json::json; let mut attrs = Vec::new(); // String attrs.push(GrowthBookAttribute::new( "email".to_string(), GrowthBookAttributeValue::String("user@example.com".to_string()) )); // Number (integer) attrs.push(GrowthBookAttribute::new( "age".to_string(), GrowthBookAttributeValue::Number(serde_json::Number::from(25)) )); // Boolean attrs.push(GrowthBookAttribute::new( "isLoggedIn".to_string(), GrowthBookAttributeValue::Bool(true) )); // Array attrs.push(GrowthBookAttribute::new( "tags".to_string(), GrowthBookAttributeValue::Array(vec![ json!("premium"), json!("beta-tester") ]) )); // Object attrs.push(GrowthBookAttribute::new( "company".to_string(), GrowthBookAttributeValue::Object(json!({ "id": "company-123", "name": "Acme Corp" })) )); ``` ### Common Attribute Patterns ```rust // Web application attributes let mut web_attrs = Vec::new(); web_attrs.push(GrowthBookAttribute::new("id".to_string(), GrowthBookAttributeValue::String(user_id))); web_attrs.push(GrowthBookAttribute::new("url".to_string(), GrowthBookAttributeValue::String(request.uri().to_string()))); web_attrs.push(GrowthBookAttribute::new("userAgent".to_string(), GrowthBookAttributeValue::String(user_agent))); web_attrs.push(GrowthBookAttribute::new("country".to_string(), GrowthBookAttributeValue::String(geo_ip_country))); // API attributes let mut api_attrs = Vec::new(); api_attrs.push(GrowthBookAttribute::new("apiKey".to_string(), GrowthBookAttributeValue::String(api_key))); api_attrs.push(GrowthBookAttribute::new("requestsToday".to_string(), GrowthBookAttributeValue::Number(serde_json::Number::from(request_count)))); api_attrs.push(GrowthBookAttribute::new("tier".to_string(), GrowthBookAttributeValue::String("premium".to_string()))); ``` ### Attribute Merging Behavior When you provide per-evaluation attributes: 1. They are **merged** with global attributes 2. Per-evaluation attributes **take precedence** over global ones 3. This allows you to set common attributes globally and override them as needed ```rust // Global attributes let mut global = HashMap::new(); global.insert("tenantId".to_string(), GrowthBookAttributeValue::String("tenant-1".to_string())); global.insert("plan".to_string(), GrowthBookAttributeValue::String("free".to_string())); let client = GrowthBookClientBuilder::new() // ... other settings ... .attributes(global) .build() .await?; // Per-evaluation attributes (overrides "plan") let mut user_attrs = Vec::new(); user_attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String("user-123".to_string()) )); user_attrs.push(GrowthBookAttribute::new( "plan".to_string(), // This overrides the global "plan" GrowthBookAttributeValue::String("premium".to_string()) )); // Final attributes used: { tenantId: "tenant-1", plan: "premium", userId: "user-123" } let result = client.is_on("premium-feature", Some(user_attrs)); ``` ## Using Features The SDK provides multiple methods for evaluating features with different levels of detail. ### Basic Feature Checks #### `is_on()` - Simple Boolean Check Check if a feature is enabled (evaluates to a truthy value): ```rust // Without attributes if client.is_on("new-navigation", None) { println!("Show new navigation"); } // With attributes let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String("user-123".to_string()) )); if client.is_on("beta-feature", Some(attrs)) { println!("User is in beta program"); } ``` #### `is_off()` - Inverse Boolean Check Check if a feature is disabled (evaluates to a falsy value): ```rust if client.is_off("maintenance-mode", None) { // Allow normal operations process_request(); } ``` ### Getting Feature Values #### `feature_result()` - Get Detailed Feature Information Get the full feature result with metadata: ```rust let result = client.feature_result("button-color", None); // Access the raw value println!("Value: {:?}", result.value); // Check if enabled if result.on { println!("Feature is on"); } // Get typed value with error handling match result.value_as::() { Ok(color) => println!("Button color: {}", color), Err(e) => println!("Error getting value: {}", e), } // Understand why this value was assigned println!("Source: {:?}", result.source); // e.g., "experiment", "force", "defaultValue" ``` ### Type-Safe Feature Values The `value_as::()` method provides type-safe access to feature values: ```rust // String values let color_result = client.feature_result("button-color", None); let color: String = color_result.value_as::() .unwrap_or("blue".to_string()); // Integer values let max_items_result = client.feature_result("max-items", None); let max_items: i32 = max_items_result.value_as::() .unwrap_or(10); // Boolean values let enabled_result = client.feature_result("new-feature", None); let enabled: bool = enabled_result.value_as::() .unwrap_or(false); // Complex types (JSON) use serde::{Deserialize, Serialize}; #[derive(Debug, Deserialize, Serialize)] struct ThemeConfig { primary_color: String, secondary_color: String, font_size: i32, } let theme_result = client.feature_result("theme-config", None); match theme_result.value_as::() { Ok(theme) => { println!("Primary: {}, Font: {}", theme.primary_color, theme.font_size); }, Err(_) => { // Use default theme let default_theme = ThemeConfig { primary_color: "blue".to_string(), secondary_color: "gray".to_string(), font_size: 16, }; } } ``` ### Feature Result Properties The `FeatureResult` struct contains detailed information about the feature evaluation: ```rust let result = client.feature_result("my-feature", None); // The actual feature value (Option) println!("Value: {:?}", result.value); // Boolean helpers if result.on { // Feature value is truthy } if result.off { // Feature value is falsy } // Why was this value assigned? match result.source { FeatureResultSource::DefaultValue => println!("Using default value"), FeatureResultSource::Force => println!("Forced value from targeting rule"), FeatureResultSource::Experiment => println!("Value from A/B test"), FeatureResultSource::UnknownFeature => println!("Feature not found"), } // Experiment information (if from an A/B test) if let Some(exp) = &result.experiment { println!("Experiment key: {}", exp.key); } if let Some(exp_result) = &result.experiment_result { println!("Variation ID: {}", exp_result.variation_id); println!("In experiment: {}", exp_result.in_experiment); } ``` ### Handling Missing Features Features that don't exist return `None` as their value: ```rust let result = client.feature_result("non-existent-feature", None); if result.value.is_none() { println!("Feature not found, using default behavior"); // Fallback logic } // Or use value_as with a default let value = result.value_as::() .unwrap_or("default-value".to_string()); ``` ### Feature Flags Usage - Best Practices ```rust // ✅ Good: Use descriptive feature keys if client.is_on("enable-dark-mode", None) { // ... } // ✅ Good: Provide fallback values let timeout = client.feature_result("api-timeout", None) .value_as::() .unwrap_or(30); // ✅ Good: Handle errors gracefully match client.feature_result("config", None).value_as::() { Ok(config) => use_config(config), Err(_) => use_default_config(), } // ❌ Bad: Using magic values without fallbacks let timeout = client.feature_result("timeout", None) .value_as::() .unwrap(); // Panics if feature doesn't exist! // ❌ Bad: Not handling type mismatches let value = client.feature_result("my-feature", None).value; // Assuming type without checking ``` ## Tracking Callbacks Tracking callbacks allow you to integrate GrowthBook with your analytics systems (Segment, Mixpanel, Amplitude, etc.) to track when users are exposed to experiments. ### Experiment Viewed Callback This callback fires when a user is assigned a variation in an A/B test: ```rust use growthbook_rust::client::GrowthBookClientBuilder; let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .on_experiment_viewed(Box::new(|experiment_result| { // Track in your analytics system println!("🧪 Experiment Viewed:"); println!(" Experiment: {}", experiment_result.key); println!(" Variation: {}", experiment_result.variation_id); println!(" Value: {:?}", experiment_result.value); println!(" In Experiment: {}", experiment_result.in_experiment); println!(" Hash Used: {}", experiment_result.hash_used); // Example: Send to Segment // analytics.track("Experiment Viewed", json!({ // "experiment_id": experiment_result.key, // "variation_id": experiment_result.variation_id, // "variation_value": experiment_result.value, // })); })) .build() .await?; ``` #### When is the Callback Triggered? The `on_experiment_viewed` callback is called when: - A feature evaluation runs an experiment - The user is included in the experiment (passes targeting rules) - The user is randomly assigned a variation (not forced) It is **NOT** called when: - A feature uses a forced value (no experiment) - The user is excluded from the experiment due to targeting - The feature doesn't exist ### Feature Usage Callback Track every feature evaluation, regardless of whether it's part of an experiment: ```rust let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .on_feature_usage(Box::new(|feature_key, result| { // Track feature usage println!("📊 Feature Used:"); println!(" Key: {}", feature_key); println!(" Value: {:?}", result.value); println!(" On: {}", result.on); println!(" Source: {:?}", result.source); // Example: Send to monitoring system // monitoring.record_metric("feature.used", 1, vec![ // format!("feature:{}", feature_key), // format!("enabled:{}", result.on), // ]); })) .build() .await?; ``` **Use Cases for Feature Usage Tracking:** - Monitor which features are being evaluated - Debug feature flag behavior - Track adoption of new features - Send metrics to monitoring systems (DataDog, New Relic) ### Using Both Callbacks Together You can use both callbacks for comprehensive tracking: ```rust let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) // Track A/B test exposures for analytics .on_experiment_viewed(Box::new(|exp_result| { // Critical for experiment analysis analytics::track_experiment_viewed( exp_result.key.clone(), exp_result.variation_id, exp_result.value.clone(), ); })) // Track all feature usage for monitoring .on_feature_usage(Box::new(|key, result| { // For debugging and monitoring monitoring::record_feature_usage(key, result.on); })) .build() .await?; ``` ### Integration Examples #### Segment Integration ```rust // Assuming you have a Segment client use segment::{HttpClient, Message}; let segment_client = HttpClient::default(); let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .on_experiment_viewed(Box::new(move |exp_result| { let message = Message::track("user-123", "Experiment Viewed") .properties(json!({ "experiment_id": exp_result.key, "variation_id": exp_result.variation_id, "variation_value": exp_result.value, })); segment_client.send(message); })) .build() .await?; ``` #### Custom Analytics System ```rust use tokio::spawn; let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .on_experiment_viewed(Box::new(|exp_result| { // Send async without blocking spawn(async move { let event = json!({ "event": "experiment_viewed", "experiment_id": exp_result.key, "variation_id": exp_result.variation_id, "timestamp": chrono::Utc::now().to_rfc3339(), }); // Send to your analytics endpoint if let Err(e) = send_analytics_event(event).await { eprintln!("Failed to send analytics: {}", e); } }); })) .build() .await?; ``` ## Context and Caching ### Context The SDK uses a context object internally to manage state. You typically don't interact with it directly, but it's useful to understand how it works: ```rust // The builder pattern creates and manages context for you let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .attributes(global_attributes) // Context attributes .build() .await?; // Per-evaluation attributes merge with context attributes let result = client.is_on("my-feature", Some(user_specific_attributes)); ``` ### Caching The SDK implements intelligent caching to minimize network requests: ```rust let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .ttl(Duration::from_secs(60)) // Cache features for 60 seconds .build() .await?; ``` **How Caching Works:** - Features are cached in memory after the first fetch - Cache is automatically refreshed based on TTL - Manual refresh with `client.refresh().await` bypasses cache - Cache is shared across all evaluations - TTL defaults to 60 seconds if not specified **Cache Behavior:** ```rust // First call: fetches from API let result1 = client.is_on("my-feature", None); // Second call within TTL: uses cache (fast) let result2 = client.is_on("my-feature", None); // After TTL expires: fetches from API again tokio::time::sleep(Duration::from_secs(61)).await; let result3 = client.is_on("my-feature", None); ``` ## Debugging and Logging ### Enable Debug Output The Rust SDK uses standard Rust logging. Enable it using `env_logger` or `tracing`: ```rust // Add to Cargo.toml // [dependencies] // env_logger = "0.11" // log = "0.4" use log::{info, debug}; fn main() { // Initialize logger env_logger::init(); // Or with custom format env_logger::Builder::from_default_env() .filter_level(log::LevelFilter::Debug) .init(); // Now SDK operations will log details let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .build() .await?; } ``` Set the log level via environment variable: ```bash RUST_LOG=debug cargo run # or RUST_LOG=growthbook_rust=debug cargo run ``` ### Common Issues and Solutions #### Issue: Features not loading ```rust // Check initialization let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .build() .await; match client { Ok(c) => println!("✅ Client initialized"), Err(e) => eprintln!("❌ Initialization failed: {}", e), } // Check if features are loaded let result = client.feature_result("test-feature", None); if result.value.is_none() { eprintln!("Feature not found - features may not be loaded"); } ``` #### Issue: Wrong feature values ```rust // Debug feature evaluation let result = client.feature_result("my-feature", None); println!("Feature: my-feature"); println!(" Value: {:?}", result.value); println!(" Source: {:?}", result.source); println!(" On: {}", result.on); // Check attributes being used let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String("test-123".to_string()) )); let result_with_attrs = client.feature_result("my-feature", Some(attrs)); println!("With attributes: {:?}", result_with_attrs.value); ``` #### Issue: Auto-refresh not working ```rust use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; let refresh_count = Arc::new(AtomicUsize::new(0)); let counter = refresh_count.clone(); let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .auto_refresh(true) .refresh_interval(Duration::from_secs(10)) .add_on_refresh(Box::new(move || { let count = counter.fetch_add(1, Ordering::SeqCst); println!("Refresh #{} at {:?}", count + 1, std::time::SystemTime::now()); })) .build() .await?; // Wait and watch for refreshes tokio::time::sleep(Duration::from_secs(35)).await; println!("Total refreshes: {}", refresh_count.load(Ordering::SeqCst)); ``` ### Testing and QA #### Testing with Forced Values ```rust // In your test environment, use features_json to control values use serde_json::json; let test_features = json!({ "feature-under-test": { "defaultValue": true }, "config-value": { "defaultValue": "test-mode" } }); let client = GrowthBookClientBuilder::new() .api_url("http://localhost:8080".to_string()) .client_key("test-key".to_string()) .features_json(test_features)? .auto_refresh(false) // Disable refresh in tests .build() .await?; // Now you can test with predictable feature values assert!(client.is_on("feature-under-test", None)); ``` #### Unit Testing Helpers ```rust #[cfg(test)] mod tests { use super::*; async fn create_test_client() -> GrowthBookClient { let features = json!({ "test-feature": { "defaultValue": true } }); GrowthBookClientBuilder::new() .api_url("http://test".to_string()) .client_key("test".to_string()) .features_json(features).unwrap() .auto_refresh(false) .build() .await .unwrap() } #[tokio::test] async fn test_feature_enabled() { let client = create_test_client().await; assert!(client.is_on("test-feature", None)); } #[tokio::test] async fn test_with_attributes() { let client = create_test_client().await; let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String("test-user".to_string()) )); let result = client.is_on("test-feature", Some(attrs)); assert!(result); } } ``` ## Integration Examples This section provides real-world integration examples with popular Rust frameworks. ### Actix Web Integration A complete example of integrating GrowthBook with Actix Web: ```rust use actix_web::{web, App, HttpRequest, HttpResponse, HttpServer}; use growthbook_rust::client::{GrowthBookClient, GrowthBookClientBuilder}; use growthbook_rust::model_public::{GrowthBookAttribute, GrowthBookAttributeValue}; use std::sync::Arc; use std::time::Duration; // Application state struct AppState { gb_client: Arc, } #[actix_web::main] async fn main() -> std::io::Result { env_logger::init(); // Initialize GrowthBook client (singleton) let gb_client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .auto_refresh(true) .refresh_interval(Duration::from_secs(30)) .on_experiment_viewed(Box::new(|exp_result| { log::info!("Experiment viewed: {} -> {}", exp_result.key, exp_result.variation_id); })) .build() .await .expect("Failed to initialize GrowthBook"); let gb_client = Arc::new(gb_client); // Start HTTP server HttpServer::new(move || { App::new() .app_data(web::Data::new(AppState { gb_client: gb_client.clone(), })) .route("/", web::get().to(index)) .route("/api/features", web::get().to(get_features)) }) .bind(("127.0.0.1", 8080))? .run() .await } // Handler with per-request attributes async fn index( data: web::Data, req: HttpRequest, ) -> HttpResponse { // Extract user attributes from request let user_id = req.headers() .get("X-User-ID") .and_then(|v| v.to_str().ok()) .unwrap_or("anonymous"); let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String(user_id.to_string()) )); attrs.push(GrowthBookAttribute::new( "url".to_string(), GrowthBookAttributeValue::String(req.uri().to_string()) )); // Check features with user-specific attributes let show_new_ui = data.gb_client.is_on("new-ui", Some(attrs.clone())); let theme_result = data.gb_client.feature_result("theme", Some(attrs)); let theme = theme_result.value_as::() .unwrap_or("light".to_string()); HttpResponse::Ok().json(serde_json::json!({ "new_ui": show_new_ui, "theme": theme, "user_id": user_id, })) } // Handler to inspect all features (useful for debugging) async fn get_features(data: web::Data) -> HttpResponse { // Return feature flags for debugging HttpResponse::Ok().json(serde_json::json!({ "status": "ok", "message": "Features loaded" })) } ``` ### Axum Integration Modern async web framework integration: ```rust use axum::{ extract::{Extension, Path}, http::{Request, StatusCode}, response::{IntoResponse, Json}, routing::{get, post}, Router, }; use growthbook_rust::client::{GrowthBookClient, GrowthBookClientBuilder}; use growthbook_rust::model_public::{GrowthBookAttribute, GrowthBookAttributeValue}; use serde::{Deserialize, Serialize}; use std::net::SocketAddr; use std::sync::Arc; use std::time::Duration; #[derive(Clone)] struct AppState { gb_client: Arc, } #[tokio::main] async fn main() { // Initialize tracing tracing_subscriber::fmt::init(); // Create GrowthBook client let gb_client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .auto_refresh(true) .refresh_interval(Duration::from_secs(30)) .build() .await .expect("Failed to create GrowthBook client"); let state = AppState { gb_client: Arc::new(gb_client), }; // Build router let app = Router::new() .route("/", get(root)) .route("/user/:id", get(user_features)) .route("/experiment", post(run_experiment)) .layer(Extension(state)); // Run server let addr = SocketAddr::from(([127, 0, 0, 1], 3000)); tracing::info!("Listening on {}", addr); axum::Server::bind(&addr) .serve(app.into_make_service()) .await .unwrap(); } async fn root(Extension(state): Extension) -> impl IntoResponse { let is_maintenance = state.gb_client.is_on("maintenance-mode", None); if is_maintenance { return ( StatusCode::SERVICE_UNAVAILABLE, "Maintenance mode enabled", ).into_response(); } (StatusCode::OK, "Service operational").into_response() } #[derive(Debug, Serialize)] struct UserFeatures { user_id: String, premium_enabled: bool, theme: String, max_uploads: i32, } async fn user_features( Path(user_id): Path, Extension(state): Extension, ) -> Json { // Build user attributes let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String(user_id.clone()) )); // Evaluate features for this user let premium_enabled = state.gb_client.is_on("premium-features", Some(attrs.clone())); let theme_result = state.gb_client.feature_result("theme", Some(attrs.clone())); let theme = theme_result.value_as::() .unwrap_or("default".to_string()); let uploads_result = state.gb_client.feature_result("max-uploads", Some(attrs)); let max_uploads = uploads_result.value_as::() .unwrap_or(5); Json(UserFeatures { user_id, premium_enabled, theme, max_uploads, }) } #[derive(Debug, Deserialize)] struct ExperimentRequest { user_id: String, feature_key: String, } async fn run_experiment( Extension(state): Extension, Json(payload): Json, ) -> impl IntoResponse { let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String(payload.user_id.clone()) )); let result = state.gb_client.feature_result(&payload.feature_key, Some(attrs)); Json(serde_json::json!({ "feature_key": payload.feature_key, "user_id": payload.user_id, "value": result.value, "on": result.on, "source": format!("{:?}", result.source), })) } ``` ### Rocket Integration ```rust #[macro_use] extern crate rocket; use rocket::{State, http::Status}; use rocket::serde::json::Json; use growthbook_rust::client::{GrowthBookClient, GrowthBookClientBuilder}; use growthbook_rust::model_public::{GrowthBookAttribute, GrowthBookAttributeValue}; use std::sync::Arc; use std::time::Duration; struct GBState { client: Arc, } #[get("/")] async fn index(state: &State) -> Result { if state.client.is_on("maintenance-mode", None) { return Err(Status::ServiceUnavailable); } Ok("Hello, World!".to_string()) } #[get("/feature/")] async fn check_feature( user_id: String, state: &State, ) -> Json { let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String(user_id.clone()) )); let enabled = state.client.is_on("beta-feature", Some(attrs)); Json(serde_json::json!({ "user_id": user_id, "beta_enabled": enabled, })) } #[launch] async fn rocket() -> _ { let gb_client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .auto_refresh(true) .build() .await .expect("Failed to init GrowthBook"); rocket::build() .manage(GBState { client: Arc::new(gb_client), }) .mount("/", routes![index, check_feature]) } ``` ### CLI Application Example Using GrowthBook in a command-line application: ```rust use growthbook_rust::client::GrowthBookClientBuilder; use growthbook_rust::model_public::{GrowthBookAttribute, GrowthBookAttributeValue}; use clap::Parser; use std::time::Duration; #[derive(Parser, Debug)] #[clap(author, version, about, long_about = None)] struct Args { /// User ID #[clap(short, long)] user_id: String, /// Command to run #[clap(short, long)] command: String, } #[tokio::main] async fn main() -> Result> { env_logger::init(); let args = Args::parse(); // Initialize GrowthBook let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .auto_refresh(false) // No auto-refresh for CLI .build() .await?; // Create user attributes let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String(args.user_id.clone()) )); attrs.push(GrowthBookAttribute::new( "cli".to_string(), GrowthBookAttributeValue::Bool(true) )); // Check feature flags let can_use_beta_commands = client.is_on("cli-beta-commands", Some(attrs.clone())); match args.command.as_str() { "beta-feature" if can_use_beta_commands => { println!("✅ Beta feature enabled for user {}", args.user_id); run_beta_feature(); } "beta-feature" => { println!("❌ Beta feature not available for user {}", args.user_id); } "standard" => { println!("Running standard command"); run_standard_command(); } _ => { println!("Unknown command: {}", args.command); } } Ok(()) } fn run_beta_feature() { println!("Executing beta feature..."); // Beta feature logic } fn run_standard_command() { println!("Executing standard command..."); // Standard logic } ``` ### Background Worker / Job Processor Using GrowthBook in async background workers: ```rust use growthbook_rust::client::{GrowthBookClient, GrowthBookClientBuilder}; use growthbook_rust::model_public::{GrowthBookAttribute, GrowthBookAttributeValue}; use std::sync::Arc; use std::time::Duration; use tokio::time::sleep; struct JobProcessor { gb_client: Arc, } impl JobProcessor { async fn new() -> Result> { let gb_client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .auto_refresh(true) .refresh_interval(Duration::from_secs(60)) .build() .await?; Ok(Self { gb_client: Arc::new(gb_client), }) } async fn process_job(&self, job: Job) { // Create attributes for this job let mut attrs = Vec::new(); attrs.push(GrowthBookAttribute::new( "userId".to_string(), GrowthBookAttributeValue::String(job.user_id.clone()) )); attrs.push(GrowthBookAttribute::new( "jobType".to_string(), GrowthBookAttributeValue::String(job.job_type.clone()) )); // Check if new processing algorithm is enabled let use_new_algorithm = self.gb_client.is_on( "new-job-processing", Some(attrs.clone()) ); if use_new_algorithm { log::info!("Using new processing algorithm for job {}", job.id); self.process_with_new_algorithm(&job).await; } else { log::info!("Using standard processing for job {}", job.id); self.process_with_standard_algorithm(&job).await; } // Check rate limits from feature flags let rate_limit_result = self.gb_client.feature_result( "job-rate-limit", Some(attrs) ); if let Ok(rate_limit) = rate_limit_result.value_as::() { log::info!("Rate limit for this job: {}", rate_limit); // Apply rate limiting } } async fn process_with_new_algorithm(&self, job: &Job) { // New algorithm logic log::info!("Processing job {} with new algorithm", job.id); } async fn process_with_standard_algorithm(&self, job: &Job) { // Standard logic log::info!("Processing job {} with standard algorithm", job.id); } } #[derive(Debug)] struct Job { id: String, user_id: String, job_type: String, } #[tokio::main] async fn main() -> Result> { env_logger::init(); let processor = JobProcessor::new().await?; // Simulate job queue loop { // Fetch job from queue (Redis, RabbitMQ, etc.) let job = Job { id: uuid::Uuid::new_v4().to_string(), user_id: "user-123".to_string(), job_type: "data-processing".to_string(), }; processor.process_job(job).await; sleep(Duration::from_secs(5)).await; } } ``` ## Advanced Usage ### Multiple SDK Instances You can create multiple GrowthBook clients for different environments or projects: ```rust use std::collections::HashMap; struct MultiTenantGrowthBook { clients: HashMap>, } impl MultiTenantGrowthBook { async fn new(tenants: Vec) -> Result> { let mut clients = HashMap::new(); for (tenant_id, api_url, client_key) in tenants { let client = GrowthBookClientBuilder::new() .api_url(api_url.to_string()) .client_key(client_key.to_string()) .auto_refresh(true) .build() .await?; clients.insert(tenant_id.to_string(), Arc::new(client)); } Ok(Self { clients }) } fn get_client(&self, tenant_id: &str) -> Option> { self.clients.get(tenant_id) } } // Usage let multi_tenant = MultiTenantGrowthBook::new(vec![ ("tenant-a", "https://cdn.growthbook.io", "sdk-key-a"), ("tenant-b", "https://cdn.growthbook.io", "sdk-key-b"), ]).await?; if let Some(client) = multi_tenant.get_client("tenant-a") { let enabled = client.is_on("feature", None); } ``` ## TypeScript / Rust Interop If you're building a hybrid application with TypeScript frontend and Rust backend, you can use the same SDK concepts across both: **Rust Backend:** ```rust // backend/src/main.rs let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .build() .await?; let features_for_frontend = serde_json::json!({ "dark_mode": client.is_on("dark-mode", Some(user_attrs)), "theme": client.feature_result("theme", Some(user_attrs)).value, }); // Send to frontend HttpResponse::Ok().json(features_for_frontend) ``` **TypeScript Frontend:** ```typescript // frontend/src/features.ts interface Features { dark_mode: boolean; theme: string; } const features: Features = await fetch('/api/features').then(r => r.json()); if (features.dark_mode) { enableDarkMode(); } ``` ## Performance Considerations ### Memory Usage - Each client instance maintains an in-memory cache of features - Auto-refresh spawns a background task - Features are deserialized from JSON on each fetch **Optimization Tips:** ```rust // ✅ Good: Single client instance, reused across requests lazy_static! { static ref GB_CLIENT: Arc = { // Initialize once at startup tokio::runtime::Runtime::new() .unwrap() .block_on(async { Arc::new( GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .build() .await .expect("Failed to create GrowthBook client") ) }) }; } // ❌ Bad: Creating new client on each request async fn handler() { let client = GrowthBookClientBuilder::new() // DON'T DO THIS .build() .await .unwrap(); } ``` ### Network Performance - Features are cached based on TTL - Consider longer refresh intervals for stable features ```rust // For frequently changing features let client = GrowthBookClientBuilder::new() .refresh_interval(Duration::from_secs(30)) // 30 seconds .build() .await?; // For stable features let client = GrowthBookClientBuilder::new() .refresh_interval(Duration::from_secs(300)) // 5 minutes .build() .await?; ``` ## Troubleshooting ### Common Error Messages #### "Failed to fetch features" ```rust // Possible causes: // 1. Wrong API URL or client key // 2. Network connectivity issues // 3. API is down // Solution: Check configuration and network let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) // Verify this .client_key("sdk-abc123".to_string()) // Verify this .build() .await .map_err(|e| { eprintln!("Client creation failed: {}", e); e })?; ``` #### "Decryption failed" ```rust // Cause: Wrong decryption key // Solution: Verify the key from GrowthBook dashboard let correct_key = env::var("GROWTHBOOK_DECRYPTION_KEY")?; let client = GrowthBookClientBuilder::new() .decryption_key(correct_key) .build() .await?; ``` #### "Feature not found" ```rust let result = client.feature_result("my-feature", None); if result.value.is_none() { // Feature doesn't exist in your GrowthBook project // Check the feature key spelling eprintln!("Feature 'my-feature' not found"); } ``` ### Enable Verbose Logging ```bash # Maximum verbosity RUST_LOG=trace cargo run # GrowthBook-specific logs RUST_LOG=growthbook_rust=debug cargo run # Filter by module RUST_LOG=growthbook_rust::client=debug cargo run ``` ### Health Check Endpoint ```rust use axum::{Extension, Json}; async fn health_check( Extension(state): Extension, ) -> Json { // Check if GrowthBook is accessible let test_result = state.gb_client.feature_result("health-check", None); Json(serde_json::json!({ "status": "ok", "growthbook": { "initialized": true, "features_loaded": !test_result.value.is_none(), } })) } ``` ## Migration from Other SDKs ### From Node.js/JavaScript SDK **JavaScript:** ```javascript const gb = new GrowthBook({ apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: { userId: "123" } }); await gb.init(); const enabled = gb.isOn("my-feature"); ``` **Rust:** ```rust let mut attrs = HashMap::new(); attrs.insert("userId".to_string(), GrowthBookAttributeValue::String("123".to_string())); let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .attributes(attrs) .build() .await?; let enabled = client.is_on("my-feature", None); ``` ### From Python SDK **Python:** ```python gb = GrowthBook( api_host="https://cdn.growthbook.io", client_key="sdk-abc123", attributes={"userId": "123"} ) gb.load_features() enabled = gb.is_on("my-feature") ``` **Rust:** ```rust let mut attrs = HashMap::new(); attrs.insert("userId".to_string(), GrowthBookAttributeValue::String("123".to_string())); let client = GrowthBookClientBuilder::new() .api_url("https://cdn.growthbook.io".to_string()) .client_key("sdk-abc123".to_string()) .attributes(attrs) .build() .await?; let enabled = client.is_on("my-feature", None); ``` ## Further Reading - [Rust SDK on crates.io ](https://crates.io/crates/growthbook-rust) - [API Documentation ](https://docs.rs/growthbook-rust) - [GitHub Repository ](https://github.com/growthbook/growthbook-rust) - [Example Application ](https://github.com/growthbook/growthbook-rust/tree/main/examples/client) ## Supported Features --- ## HTML Script Tag # HTML Script Tag SDK We provide an HTML `` tag option for easily integrating GrowthBook into any website. This option is quick and straightforward to use and does not require coding knowledge to implement. ## Installation For this, you will need a **Client Key** from an SDK Connection within GrowthBook. This will start with `sdk-`. Add the following script tag to your website and replace `YOUR_CLIENT_KEY_HERE` with your actual Client Key: ```html ``` Or, if you can't add `data-*` attributes (e.g. in Google Tag Manager), you can use this alternate version: ```html (function(s) { s=document.createElement('script'); s.async=true; s.dataset.clientKey="YOUR_CLIENT_KEY_HERE"; s.src="https://cdn.jsdelivr.net/npm/@growthbook/growthbook/dist/bundles/auto.min.js"; document.head.appendChild(s); })(); ``` For best performance, we recommend adding this script tag directly before the closing `` tag, however it is also possible to load through something like Google Tag Manager if that is your only option. ### Optional Configuration Settings Besides the Client Key, there are some additional settings you can define with `data-` attributes on the script tag: - `data-api-host` - Defaults to GrowthBook Cloud's https://cdn.growthbook.io. This must be set to your own domain instead if self-hosting or using the GrowthBook Proxy. - `data-decryption-key` - Required if you enabled encryption in your SDK Connection settings in GrowthBook All other settings can be defined with a `window.growthbook_config` object BEFORE you load the script tag. This accepts the same settings as our [JavaScript SDK](/lib/js). Here's an example: ```html window.growthbook_config = window.growthbook_config || {}; // Disable streaming updates window.growthbook_config.backgroundSync = false; ``` If you need low level access to the GrowthBook SDK instance for any reason, you can use a callback function. This will be called as soon as the GrowthBook SDK instance is ready. If you push to the queue after GrowthBook has already loaded, the callback will be invoked immediately. ```html window.growthbook_queue = window.growthbook_queue || []; window.growthbook_queue.push((gb) => { // Do whatever you need with the GrowthBook instance here console.log(gb.getAttributes()); }) ``` ## Targeting Attributes The following targeting attributes are set automatically and available for use. - `id` - creates a long-lived `gbuuid` cookie if it doesn't exist already - `url` - `path` - `host` - `query` - `pageTitle` - `deviceType` - either `mobile` or `desktop` - `browser` - one of `chrome`, `edge`, `firefox`, `safari`, or `unknown` - `utmSource` - `utmMedium` - `utmCampaign` - `utmTerm` - `utmContent` In addition, if you use Google Tag Manager, any variables you set in your Data Layer will also be set here and available for targeting. ### Adding Custom Attributes You can include your own custom attributes by adding the following BEFORE the GrowthBook snippet: ```html window.growthbook_config = window.growthbook_config || {}; window.growthbook_config.attributes = { country: "US", otherCustomAttribute: 12, } ``` You can also set custom attributes later, after the script tag has been added. Please note, this may cause flickering in your experiments if you reference these custom attributes in your experiment targeting. ```html window.growthbook_queue = window.growthbook_queue || []; window.growthbook_queue.push((gb) => { gb.updateAttributes({ country: "US", otherCustomAttribute: 12, }); }); ``` ### Refreshing Auto Attributes The GrowthBook snippet will automatically watch for URL changes and update attributes when that happens. If you would like more control over this behavior, you can manually trigger updates at any time by firing a `growthbookrefresh` event from JavaScript. For example: ```ts document.dispatchEvent(new CustomEvent("growthbookrefresh")); ``` ## Tracking Experiment Views When a user views an experiment, a tracking event is fired. There is built-in support for Segment.io (`analytics.js`), GA4 (`gtag`), and Google Tag Manager (`dataLayer`). ### Segment.io GrowthBook will automatically fire an event using `window.analytics.track` if it's present on the page. The event name is `Experiment Viewed` and it has two properties: `experiment_id` and `variation_id`. There is no additional configuration needed. ### GA4 (`gtag`) GrowthBook will automatically fire an event using Google Analytic 4's `window.gtag` if it's present on the page. The event name is `experiment_viewed` and it has two properties: `experiment_id` and `variation_id`. There is no additional configuration needed. ### Google Tag Manager We send the following event to the Data Layer. You will need to add a trigger based on this and forward it on to your analytics tool of choice. ```json { "event": "experiment_viewed", "experiment_id": "...", "variation_id": "..." } ``` We have a walkthrough tutorial on how to configure this in our [GTM Guide](/guide/google-tag-manager-and-growthbook#4-tracking-via-datalayer-and-gtm) ### Mixpanel To work with Mixpanel, we have to set the ID as an attribute for GrowthBook, and also add the custom event tracking callback. Below is an example of how to set up GrowthBook with Mixpanel. The script initializes Mixpanel (which you probably already have) and then sets the Mixpanel distinct ID to a GrowthBook `id` attribute once it has loaded. Then the script defines a trackingCallback to log the experiment exposure event to Mixpanel. ```html window.growthbook_config = window.growthbook_config || {}; mixpanel.init("[YOUR PROJECT TOKEN]", { debug: true, loaded: function (mx) { window.growthbook_queue = window.growthbook_queue || []; window.growthbook_queue.push((gb) => { gb.setAttributes({ ...gb.getAttributes(), id: mx.get_distinct_id(), }); }) }, }); window.growthbook_config.trackingCallback = (experiment, result, user) => { mixpanel.track("$experiment_started", { "Experiment name": experiment.key, "Variant name": result.variationId, "GrowthBook attributes": user?.attributes, $source: "growthbook", }); ``` ### Others For any other event tracking system (or if the built-in ones above do not meet your requirements), you can define your own custom tracking callback function. This must be defined _BEFORE_ loading the main GrowthBook snippet on the page. Starting in version 1.7.0, the callback receives a third `user` argument containing the `attributes` and optional `url` used when the experiment was evaluated. We recommend recording attributes from this argument so exposure events consistently reflect the attributes GrowthBook used for targeting and assignment. ```html window.growthbook_config = window.growthbook_config || {}; window.growthbook_config.trackingCallback = (experiment, result, user) => { customEventTracker("Viewed Experiment", { experiment_id: experiment.key, variation_id: result.key, attributes: user?.attributes }) }; ``` ## Sticky Bucketing This SDK supports [Sticky Bucketing](/app/sticky-bucketing), but it is disabled by default. To enable, add the following to your script tag: ``` data-use-sticky-bucket-service="cookie" ``` There are 2 possible values: - **cookie** - Persist in a cookie that is shared between the browser and your server - **localStorage** - Persist in the browser's localStorage, which is never sent to your server You can also customize the key we use to store sticky buckets by adding a prefix: ``` data-sticky-bucket-prefix="my_prefix" ``` ## Using Feature Flags You can use feature flags with this SDK, but it does require some manual coding work. Here is an example: ```html // Wait for the GrowthBook SDK to load before running window.growthbook_queue = window.growthbook_queue || []; window.growthbook_queue.push((gb) => { // Function that uses feature flags to make changes to the page const applyFeatureFlags = () => { if(gb.isOn("dark-mode")) { document.documentElement.classList.add("dark"); } else { document.documentElement.classList.remove("dark"); } } // Call your function initially plus whenever new data is received applyFeatureFlags(); document.addEventListener("growthbookdata", applyFeatureFlags) }); ``` ## Cookie Policy and GDPR By default, this SDK persists a random unique identifier in a first-party cookie named `gbuuid`. This cookie is required to provide a consistent user experience to your visitors. Without this cookie, if you run an A/B test, a visitor might be re-bucketed into a different variation every time they visit your website. The `gbuuid` cookie does not contain any Personally Identifiable Information (it's just a randomly generated id). It is a first-party cookie that is never shared with any third-party services, not even GrowthBook itself. However, we still recommend adding this to your site's cookie policy if you have one. ### Sharing the Cookie Across Subdomains By default the `gbuuid` cookie is **host-only**, so it is not readable on other subdomains. If your site spans multiple subdomains (for example `www.example.com` and `app.example.com`) — or if you run **redirect experiments that send users to another subdomain** — the visitor will get a brand new `gbuuid` on the new subdomain and be re-bucketed. This creates duplicate anonymous ids for the same person, which shows up as inflated user counts and a Sample Ratio Mismatch (SRM) in your analytics/warehouse. To keep one stable id across all subdomains, scope the cookie to your parent domain with `data-uuid-cookie-domain` (or `window.growthbook_config.uuidCookieDomain`): ```html ``` When set, this also scopes the sticky bucketing cookies (`data-use-sticky-bucket-service="cookie"`) to the same domain, so sticky assignments follow the shared id across subdomains. Note that cookies cannot be shared across completely different registrable domains (e.g. `example.com` and `example.co.uk`). For that case, pass the id through the redirect URL and initialize the destination with `data-uuid` / `window.growthbook_config.uuid`. ### Delay Storing the Cookie Until Consent is Granted If you must delay persisting the `gbuuid` cookie until a user consents, you can add a `data-no-auto-cookies` attribute to the script tag. This will still generate a UUID for the user, but will not persist it. That means, if the user refreshes the page, they will have a new random UUID generated. You have the option to manually persist this cookie at any time, for example when a user grants consent on your cookie banner. All you need to do is fire this custom event from javascript: ```js document.dispatchEvent(new CustomEvent("growthbookpersist")); ``` ## Content Security Policy (CSP) If your site uses a Content Security Policy, you may need to make a few changes to the `script-src` directive for this SDK to load and run on your site. First, make sure to add `https://cdn.jsdelivr.net` so the script itself can load. If this isn't possible, you can save the contents of the script and host it on your own domain instead. Just note that doing that means you will no longer get automatic updates when we make improvements to the script tag. Second, if you plan to use the Visual Editor to inject custom javascript into your site, you need to allow both `unsafe-inline` and `unsafe-eval`. If this isn't possible, we have an alternative using nonces (see below). ### Using Script Nonces As an alternative to allowing `unsafe-inline`, we support "nonces", although this requires some very technical and custom configuration to hook up. This is only required if you plan to use the Visual Editor to inject custom javascript into your site. You will still need to allow `unsafe-eval` due to how our Visual Editor works under-the-hood. First, you will need to generate a unique nonce value for every request and add it to your CSP header. This can be done on the edge such as with a Cloudflare Worker. Lastly, you will also need to inject the following into your page's `` BEFORE you load the GrowthBook snippet. This can be accomplished in the same edge worker. Replace all instances of `$NONCE` with the unique nonce value you generated. ```html window.growthbook_config = window.growthbook_config || {}; window.growthbook_config.jsInjectionNonce = "$NONCE"; ``` --- ## Swift SDK import SdkResources from '@site/src/components/SdkResources'; import SdkSupportedFeatures from '@site/src/components/SdkSupportedFeatures'; # Swift (iOS) This SDK supports the following platforms and versions: - iOS 12 and above - Apple TvOS 12 and above - Apple WatchOS 5.0 and above ## Technical Specifications ### Platform Support The GrowthBook Swift SDK provides comprehensive support for all Apple platforms: - **iOS**: 12.0+ (iPhone, iPad) - **macOS**: 10.15+ (Catalina and later) - **tvOS**: 12.0+ (Apple TV) - **watchOS**: 5.0+ (Apple Watch) - **visionOS**: 1.0+ (Apple Vision Pro) ### Performance Characteristics - **Lightweight**: Minimal memory footprint and fast initialization - **Offline-First**: Works without network connectivity using cached features - **Efficient Caching**: Intelligent cache management with automatic cleanup - **Background Sync**: Real-time updates via Server-Sent Events (SSE) - **Low Latency**: Sub-millisecond feature evaluation performance ### Implementation Details #### Architecture - **Native Swift**: Built entirely in Swift for optimal performance - **Protocol-Oriented**: Uses Swift protocols for extensibility and testability - **Memory Safe**: Leverages Swift's memory safety features - **Concurrent**: Thread-safe implementation with proper synchronization #### Dependencies - **No External Dependencies**: Zero third-party dependencies for maximum reliability - **Foundation Framework**: Uses only Apple's Foundation framework - **CryptoKit**: Leverages Apple's CryptoKit for encryption operations - **Network Framework**: Uses modern URLSession for networking #### Data Formats - **JSON**: Native JSON parsing and serialization - **Base64**: Standard Base64 encoding for encrypted payloads - **UTF-8**: Full Unicode support for international applications #### Network Protocol - **HTTPS**: Secure communication with GrowthBook servers - **REST API**: Standard HTTP/HTTPS for feature fetching - **SSE**: Server-Sent Events for real-time updates - **Retry Logic**: Exponential backoff for failed requests ### Memory Management - **ARC**: Automatic Reference Counting for memory management - **Weak References**: Prevents retain cycles in callbacks - **Lazy Loading**: Features are loaded on-demand to minimize memory usage - **Cache Limits**: Configurable cache size limits to prevent memory bloat ### Thread Safety - **Concurrent Access**: Safe for use across multiple threads - **Atomic Operations**: Thread-safe attribute updates and feature evaluation - **Queue Management**: Proper dispatch queue usage for background operations - **Lock-Free**: Minimal locking for optimal performance ## Installation ### CocoaPods Add the following to your podfile: ```ruby source 'https://github.com/CocoaPods/Specs.git' target 'MyApp' do pod 'GrowthBook-IOS' end ``` Then, install: ```bash pod install ``` ### Swift Package Manager (SPM) Add GrowthBook to your `Package.swift` file: ```swift dependencies: [ .package(url: "https://github.com/growthbook/growthbook-swift.git") ] ``` ## Quick Start ### 1) Configure ```swift var gb: GrowthBookSDK = GrowthBookBuilder( // Your GrowthBook API host apiHost: "https://cdn.growthbook.io", // Your client key clientKey: "sdk-abc123", // Optional: encryption key if your features are encrypted encryptionKey: "abcdef98765", // Required user attributes for targeting (can be empty initially) attributes: [:], // Required: called whenever someone is put into an experiment trackingCallback: { experiment, result in // Track experiment exposures in your analytics }, // Optional: real-time updates via SSE backgroundSync: false, // Optional: called when features are refreshed (SSE or manual refresh) refreshHandler: { success in // Update UI or trigger re-render if needed } ).initializer() ``` ### 2) Set attributes ```swift var attrs = [ "id": "12345", // stable user id for bucketing "deviceId": "abc123", // fallback identifier if user id is unavailable "loggedIn": true, "country": "US" ] gb.setAttributes(attrs) ``` ### 3) Start feature flagging ```swift if gb.isOn(feature: "feature-usage-code") { // Feature is enabled! } import Foundation // For JSON type let value = gb.getFeatureValue(feature: "button-color", default: JSON("blue")) ``` ### 4) Wait for features to load - With background sync off, call `gb.refreshCache()` at app start or navigation points and show a loading state until complete. - With background sync on, initial cache loads synchronously if present and updates stream in via SSE. - Use `refreshHandler` to re-render when new features arrive. ```swift var gb: GrowthBookSDK = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: [:], trackingCallback: { experiment, result in // Track experiments }, // Callback when features are refreshed refreshHandler: { success in if success { print("Features refreshed successfully") } else { print("Failed to refresh features") } }, backgroundSync: true ).initializer() ``` ## Loading Features ### Built-in fetching and refresh mechanisms If you pass an `apiHost` and `clientKey` into `GrowthBookBuilder`, the SDK handles network requests, caching, retry/backoff, and decryption (when `encryptionKey` is provided). Use `gb.refreshCache()` to manually refresh on navigation or app resume. ```swift var gb: GrowthBookSDK = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: [:], trackingCallback: { _, _ in }, // Optional: decrypt encrypted payloads encryptionKey: "abcdef98765", // Optional: real-time updates can be enabled later backgroundSync: false ).initializer() // Refresh on demand (e.g., app start or navigation) _ = gb.refreshCache() ``` ### Secure requests: custom HTTP headers Add custom headers for API requests (e.g., auth tokens, custom user-agent). Provide default headers for the standard feature fetches and separate headers for SSE streaming if needed. ```swift var gb: GrowthBookSDK = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: [:], trackingCallback: { _, _ in } ) // Sent with feature fetch requests to the apiHost .setApiHostRequestHeaders([ "Authorization": "Bearer ", "X-App-Version": "1.2.3" ]) // Sent with SSE connections to the streaming host .setStreamingHostRequestHeaders([ "Authorization": "Bearer ", "X-Device": "iPhone15,3" ]) .initializer() ``` ### Streaming updates (SSE) Enable `backgroundSync: true` to receive real-time updates via SSE. The SDK reconnects automatically when network conditions change. Use `refreshHandler` to update UI when features change. ```swift var gb: GrowthBookSDK = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: [:], trackingCallback: { _, _ in }, refreshHandler: { success in // Trigger UI updates when features change }, backgroundSync: true ).initializer() ``` Monitor the SSE lifecycle using your app logs/telemetry. On transient network errors, the SDK will attempt automatic reconnects with backoff. #### SSE resume support (Last-Event-Id) When using `backgroundSync: true`, the SDK connects via SSE. To resume after interruptions without duplicating events, you can provide the last received event id so the server continues from that point. ```swift // Persist and restore lastEventId yourself (e.g., UserDefaults) if you need fine-grained control let lastEventId = UserDefaults.standard.string(forKey: "gb_last_event_id") var gb: GrowthBookSDK = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: [:], trackingCallback: { _, _ in }, backgroundSync: true ) .setStreamingHostRequestHeaders(["Last-Event-Id": lastEventId ?? ""]) // optional .initializer() // Somewhere in your SSE handler, update lastEventId when new events arrive // UserDefaults.standard.set(newEventId, forKey: "gb_last_event_id") ``` ## Caching The SDK persists downloaded feature payloads and related metadata to disk. By default, it will: - Use a platform-specific system directory - iOS/watchOS/macOS: **Application Support** (`.applicationSupport`) - tvOS: **Caches** (`.caches`), since Application Support is not available on tvOS - Create the cache folder automatically on first use - Isolate data per `clientKey` by nesting under a client-specific subfolder - Generate deterministic cache file names by hashing keys with **SHA256** and prefixing file names with the first 5 characters of the hash for readability You can fully customize where and how the SDK caches data: ```swift // Cache configuration var gb: GrowthBookSDK = GrowthBookBuilder(...) // Choose a system directory (defaults shown below) .setSystemCacheDirectory( // Default: .applicationSupport for iOS/watchOS/macOS; .caches for tvOS #if os(tvOS) .caches #else .applicationSupport #endif ) // Or, provide an absolute custom path. The SDK will create it if missing. .setCustomCacheDirectory("/path/to/custom/cache") .initializer() // Cache management // Removes all on-disk cache for this SDK instance (scoped to the configured client key) gb.clearCache() ``` :::note Cache management - The cache directory will be created automatically if it does not exist. - Cache file keys are SHA256-hashed with a 5-character prefix for human-friendly diagnostics while avoiding path-length and special-character issues. - Multiple environments/apps can safely share the same parent directory; per-`clientKey` subfolders prevent collisions. ::: #### Cache Key Hashing The SDK uses **SHA-256** for generating deterministic cache file names: - **Purpose**: Create unique, collision-resistant cache file identifiers - **Format**: First 5 characters of SHA-256 hash for human-readable diagnostics - **Isolation**: Per-client-key subfolders prevent cross-environment collisions ### Custom integration (local evaluation) If you prefer to control network and caching yourself, pass features as a JSON `Data` blob into the builder. This enables fully local evaluation and works offline-first. ```swift // Create JSON data for features let featuresJSON = """ { "feature1": { "defaultValue": true } } """ let featuresData = featuresJSON.data(using: .utf8)! // Initialize with features data (local evaluation) var gb: GrowthBookSDK = GrowthBookBuilder( features: featuresData, attributes: [:], trackingCallback: { _, _ in }, backgroundSync: false ).initializer() ``` ## Remote evaluation vs. local evaluation :::tip See the [Remote Evaluation overview](/self-host/remote-evaluation) for more information about what Remote Evaluation is, how it works, and deployment options. ::: - Local evaluation loads a full feature payload into the SDK (e.g., via built-in fetching or a bundled JSON) and evaluates rules on-device. - Remote evaluation delegates evaluation to GrowthBook and downloads already-evaluated feature values for the current attributes. This reduces payload size and can simplify privacy controls. - Choose based on privacy, payload size, and offline requirements. Local evaluation (bundled JSON): ```swift let featuresJSON = """{ "feature1": { "defaultValue": true } }""" let featuresData = featuresJSON.data(using: .utf8)! let gbLocal = GrowthBookBuilder( features: featuresData, attributes: ["id": "12345"], trackingCallback: { _, _ in } ).initializer() ``` Remote evaluation (standard fetching): ```swift let gbRemote = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: ["id": "12345"], trackingCallback: { _, _ in } ).initializer() ``` ### Manual remote evaluation If you use remote evaluation mode, you can trigger a re-fetch of evaluated payloads when user attributes or context change without waiting for the standard refresh cadence. ```swift // When user attributes change significantly, trigger a remote eval refresh gb.setAttributes(["id": "user_123", "plan": "pro"]) gb.refreshForRemoteEval() ``` #### Custom payload structure: When using remote evaluation, ensure the attributes you send include all fields required by your targeting rules. You may also include metadata your edge or proxy expects (e.g., headers added via `setApiHostRequestHeaders`). ```swift // Attributes control what the remote evaluator returns gb.setAttributes([ "id": "user_123", "country": "US", "appVersion": "1.3.0", "plan": "pro" ]) // Then manually trigger remote evaluation gb.refreshForRemoteEval() ``` ## Encrypted features The Swift SDK uses **AES-CBC (Advanced Encryption Standard - Cipher Block Chaining)** encryption for securing feature payloads: - **Algorithm**: AES-CBC with 128-bit key length - **Key Format**: Base64-encoded encryption key - **IV (Initialization Vector)**: 16-byte random IV generated for each encryption operation - **Payload Format**: `{base64_iv}.{base64_encrypted_data}` GrowthBook uses **Server-Side Encryption** to encrypt feature payloads using AES-CBC before transmission. The Swift SDK automatically decrypts payloads when the `encryptionKey` is provided in the initialization. See below example: ```swift let gbEncrypted = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", encryptionKey: "abcdef98765", attributes: [:], trackingCallback: { _, _ in } ).initializer() ``` Decrypted features are cached **securely** on the device. If decryption fails, the SDK gracefully falls back to default feature values. :::warning Security Considerations While encryption adds a layer of security, remember that: - The decryption key is accessible within the client-side application - For highly sensitive information, consider using [Remote Evaluation](#remote-evaluation-vs-local-evaluation) instead - Encryption keys should be managed securely and rotated regularly ::: ## Sticky bucketing Sticky bucketing ensures that once a user is assigned to a variation in an experiment or feature rule configured as “Sticky,” they will continue to see the same variation on subsequent evaluations. Read more about [Sticky bucketing Feature](https://docs.growthbook.io/app/sticky-bucketing) How to enable: - In GrowthBook, enable Sticky bucketing on the experiment or on the feature rule. - In the SDK, provide a stable identity attribute (commonly `id`). If no `id` is available (e.g., logged out), use a device-scoped fallback such as `deviceId` until login. By default, when a sticky rule runs the SDK will: - Look up an existing sticky assignment for the current identity key - If found, return the same variation without re-assigning - If not found, evaluate the rule normally, persist the chosen variation locally, and return it - Persist across app restarts and feature refreshes; changing non-identity attributes (e.g., `country`) will not change the sticky result. Changing the identity (e.g., user logs in/out) causes a lookup under the new identity key ```swift gb.setAttributes([ "id": "user_123", // stable across sessions "deviceId": "abcd-efgh" // fallback when logged out ]) ``` ### Custom Sticky Bucket Service with custom storage Use this to control how and where sticky assignments are persisted. Provide a prefix to namespace data (e.g., by app or environment) and a storage backend that implements simple key/value get-set-delete semantics (e.g., `UserDefaults`, Keychain, SQLite, or your own cache layer). If not provided, the SDK uses its built-in persistent storage to remember sticky assignments. ```swift // Custom Sticky Bucket Service with custom storage let customStorage = MyCachingManager() let stickyService = StickyBucketService( prefix: "myapp_sticky_", localStorage: customStorage ) var gb: GrowthBookSDK = GrowthBookBuilder(...) .setStickyBucketService(stickyBucketService: stickyService) .initializer() ``` Tips: - Use a unique `prefix` per environment (e.g., `prod_`, `staging_`) to avoid cross-environment bleed. - To “forget” a user’s sticky assignment on logout, either switch the identity you send in `attributes` (e.g., from `id` to `deviceId`) or clear the underlying storage used by your sticky service. ## Prerequisites Flags can depend on other flags via prerequisites. If a prerequisite is not met, the dependent flag will resolve to its fallback value. Ensure attributes satisfy prerequisite rules before evaluating. Read more about [Prerequisite features](https://docs.growthbook.io/features/prerequisites) ```swift let enabled = gb.isOn(feature: "child-feature") ``` ## SemVer targeting Map your app version/build into attributes and target with semver rules (e.g., rollouts to ">=1.2.0"). ```swift import Foundation let appVersion = Bundle.main.infoDictionary?["CFBundleShortVersionString"] as? String ?? "" var attrs = gb.getAttributes() attrs["appVersion"] = appVersion gb.setAttributes(attrs) ``` ## v2 hashing The SDK supports v2 hashing for cross-SDK parity and consistent bucketing. When migrating, verify exposure stability during QA across platforms. ### Hashing Algorithms The Swift SDK implements sophisticated hashing algorithms for consistent user bucketing and secure attribute handling: #### FNV32a Hashing (Fowler-Noll-Vo) The SDK uses the **FNV32a** algorithm for user bucketing and experiment assignment: - **Algorithm**: Fowler-Noll-Vo hash function (fnv32a variant) - **Purpose**: Consistent user bucketing across all SDK platforms - **Output**: 32-bit unsigned integer converted to float between 0 and 1 - **Cross-Platform**: Ensures identical bucketing behavior across iOS, Android, Web, and server-side SDKs #### Hash Version 2 (Recommended) The SDK supports **v2 hashing** which fixes bias issues present in the original algorithm: ```swift // v2 hashing formula (unbiased) n = fnv32a(fnv32a(seed + value) + "") return (n % 10000) / 10000 ``` :::info V2 Hashing V2 Hashing is: - **Unbiased**: Eliminates parallel experiment bias. - **Higher Precision**: Uses 10,000 buckets instead of 1,000 - **Consistent**: Identical behavior across all GrowthBook SDKs ::: ## Experimentation and tracking Set `trackingCallback` to capture exposures whenever a user is assigned to a variation. Send these to your analytics provider. The callback provides `experiment.key` and `experimentResult.variationId`. #### Generic example: ```swift let gbTracking = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: [:], trackingCallback: { experiment, result in print("Viewed Experiment", experiment.key, "variation:", result.variationId) } ).initializer() ``` #### Firebase Analytics example: ```swift import FirebaseAnalytics let gbFirebase = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: [:], trackingCallback: { experiment, result in Analytics.logEvent("gb_exposure", parameters: [ "experiment_key": experiment.key, "variation_id": result.variationId ]) } ).initializer() ``` ### Experiment Result subscriptions Subscribe to the experiments to receive callbacks for every experiment run. ```swift // Subscribe to experiment results gb.subscribe { experiment, result in print("Experiment: \(experiment.key), Variation: \(result.variationId)") } // Clear all subscriptions when done gb.clearSubscriptions() ``` ## Security Features The Swift SDK implements comprehensive security measures to protect sensitive data and ensure secure feature flag evaluation: ### Secure Attributes When **secure attribute hashing** is enabled in your SDK Connection, you must manually hash sensitive attributes before passing them to the SDK. This allows you to safely target users based on sensitive data without exposing it to the client. - **Manual Hashing Required**: You must hash `secureString` or `secureString[]` attributes using SHA-256 - **Salt Protection**: Use your organization's secure attribute salt for additional security - **Privacy Compliance**: Enables targeting based on sensitive data without exposing it to the client - **GDPR/CCPA Ready**: Helps maintain compliance with data protection regulations - **Format**: `SHA256(attribute_value + organization_salt)` - **Implementation**: Use Apple's CryptoKit framework for SHA-256 hashing ```swift import CryptoKit // Helper function to hash secure attributes func hashSecureAttribute(_ value: String, salt: String) -> String { let data = (value + salt).data(using: .utf8)! let hash = SHA256.hash(data: data) return hash.compactMap { String(format: "%02x", $0) }.joined() } // Your organization's secure attribute salt (set in Organization Settings) let salt = "your_organization_salt" // Hash sensitive attributes before passing to SDK let email = "user@example.com" let phone = "+1234567890" gb.setAttributes([ "id": "user_123", "email": hashSecureAttribute(email, salt: salt), // Must be manually hashed "phone": hashSecureAttribute(phone, salt: salt), // Must be manually hashed "country": "US" // Regular attribute, not hashed ]) ``` :::note Secure attribute hashing **Important**: When secure attribute hashing is enabled, all targeting conditions in the SDK payload referencing attributes with datatype `secureString` or `secureString[]` will be anonymized via SHA-256 hashing. You must manually hash these attributes using your organization's secure attribute salt. ::: ### Remote Evaluation Security For maximum security, use **Remote Evaluation** mode: - **Server-Side Processing**: Feature evaluation happens on your secure servers - **No Client Exposure**: Targeting rules and unused variations never reach the client - **Minimal Payload**: Only evaluated results are transmitted - **Enhanced Privacy**: Sensitive business logic remains server-side ```swift // Remote evaluation for maximum security let gbSecure = GrowthBookBuilder( apiHost: "https://cdn.growthbook.io", clientKey: "sdk-abc123", attributes: ["id": "user_123"], trackingCallback: { _, _ in }, remoteEval: true // Enable remote evaluation ).initializer() ``` See the [Remote Evaluation overview](/self-host/remote-evaluation) for more information. ## Troubleshooting and debugging - Comprehensive logging - Set your app/logger to debug or trace during integration to observe evaluation decisions, cache reads/writes, and SSE state changes. - Log levels: `.debug`, `.info`, `.warning`, `.error` are supported and use `setLogLevel()` to configure the level. - Built-in OS logging integration (uses `os.Logger` on iOS 14+ and `OSLog` on older versions) - QA Mode (`setQAMode()`) functionality and use cases - Be mindful to disable verbose logs in production. - Network errors - The SDK retries transient failures with backoff when fetching or streaming. Inspect your networking layer logs and verify headers/hosts. - Confirm that custom headers are accepted by your proxy/CDN and not stripped. - SSE connection status monitoring - Use your own telemetry around `refreshHandler` or UI updates to infer connectivity. Consider persisting the last event id to resume streams as shown above. ```swift // Example: simple log wrapper func log(_ message: String) { #if DEBUG print("[GrowthBook] \(message)") #endif } ``` ## Reference View detailed docs on the [GitHub Repo](https://github.com/growthbook/growthbook-swift) ## Supported Features --- ## How to Use GrowthBook with Vue import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import ExternalLink from '@site/src/components/ExternalLink' This guide shows you how to integrate GrowthBook's JavaScript SDK into your Vue app. We provide examples using both the [Composition API](https://vuejs.org/guide/introduction.html#composition-api) and the [Options API](https://vuejs.org/guide/introduction.html#options-api). While this guide focuses on using GrowthBook in a Vue app, it's important to note that the underlying functionality is powered by the GrowthBook JavaScript SDK. To explore the full range of methods, capabilities, and customization options, check out the official [JavaScript SDK docs](/lib/js). ## Installation Add the `@growthbook/growthbook` package to your project. ```sh npm install @growthbook/growthbook ``` ```sh yarn add @growthbook/growthbook ``` ```html import { GrowthBook } from "https://unpkg.com/@growthbook/growthbook/dist/bundles/esm.min.js"; //... ``` ## Create a Provider Use Vue's `app.provide` method to make GrowthBook available to your components. In your app's entry file, usually `./src/main.ts`, add the following code: ```ts // Import the GrowthBook SDK import { GrowthBook } from '@growthbook/growthbook' // Add imports needed to create the provider import type { InjectionKey } from 'vue' import { createApp, reactive } from 'vue' import App from './App.vue' import './assets/main.css' // Create a reactive instance of GrowthBook const gbInstance = reactive( new GrowthBook({ clientKey: 'YOUR_CLIENT_KEY', attributes: { // Add user attributes here }, enableDevMode: true // Optional: Enable the Visual Editor and dev tools }) ) // Share the provider type with other components export const gbKey = Symbol('gb') as InjectionKey // Initialize GrowthBook with streaming enabled for real-time updates const initializeGrowthBook = async () => { try { await gbInstance.init({ streaming: true }) return gbInstance } catch (e) { console.error('Error initializing GrowthBook:', e) return null } } initializeGrowthBook().then((gbInstance) => { const app = createApp(App) // Provide the GrowthBook instance app.provide(gbKey, gbInstance) app.mount('#app') }) ``` ## Inject GrowthBook into a Component Next, import the `gbKey` and inject GrowthBook into your component. Below, we use the `isOn` helper to check if a feature flag is on. See additional helpers in the [JS SDK docs](/lib/js). ```ts title="component.vue" // Import Vue functions and the gbKey from main to preserve type info import { inject, ref, watch } from 'vue' import { gbKey } from '../main' // Inject the GrowthBook instance const growthbook = inject(gbKey) // Create a reactive variable to store and update the feature flag result const showBanner = ref(growthbook?.isOn('show-banner')) // Optional: Watch the feature flag for changes (requires streaming to be enabled) if (growthbook) { watch(growthbook, () => { showBanner.value = growthbook?.isOn('show-banner') }) } ``` ```ts title="component.vue" // Import Vue functions and the gbKey from main to preserve type info import { inject, watch } from 'vue' import { gbKey } from '../main' export default { // Inject the GrowthBook instance inject: { gb: { from: gbKey }, }, // Define a variable to store the feature flag result data() { return { showBanner: false } }, mounted() { // Set the showBanner variable to the feature flag value if (this.gb) { this.showBanner = this.gb.isOn('show-banner') // Optional: Watch the feature flag for changes (requires streaming to be enabled) watch(this.gb, () => { this.showBanner = this.gb.isOn('show-banner') }) } } } ``` ## Use the Feature Flag in a Template Combine Vue's `v-if` directive with the `showBanner` variable to conditionally render content based on the feature flag value. ```html title="component.vue" Now you see me! ``` ## Examples See [examples ](https://github.com/growthbook/examples/tree/b5e9138064cb565ef2c7bea30b97bdbdb797f67e/vue) of using GrowthBook with Vue's Composition and Options API. --- ## Funnel Metrics import CommercialFeature from '@site/src/components/CommercialFeature'; # Funnel Metrics Funnel Metrics measure the share of exposed units who complete a series of steps in order. One metric gives you both the end-to-end completion rate and a per-step breakdown of where units drop off, so a single row in your experiment results expands into the whole journey. A funnel metric is a [Fact Table](/app/metrics) metric like any other. You define an ordered list of steps and GrowthBook turns each step into its own conversion rate. :::note Two things named "funnel" This page covers the **funnel metric type** built on Fact Tables. It is unrelated to the [legacy denominator behavior](/app/metrics/legacy#funnel-metrics), where a binomial denominator metric acted as a filter on the numerator. Those legacy metrics should be replaced by these funnel metrics, although there are some slight differences in behavior. The [Funnel explorer](/app/product-analytics/explorer#funnels) in Product Analytics answers similar questions outside of an experiment. Funnel metrics are for experiment analysis; the explorer is for ad-hoc exploration. ::: ## Creating a funnel metric 1. Go to **Metrics** and click **Add Metric**. 2. Pick your **Data Source**, then choose **Funnel** as the metric type. 3. Select the **Fact Table** your step events come from. All steps read from this one table. 4. Name the first step and add a row filter that defines it, such as `event_name = view_product`. 5. Click **Add a step** for each additional step, in the order units should complete them. 6. Set conversion windows and mark optional steps as needed, then save. A funnel needs at least 2 steps and supports up to 20. :::info One fact table per funnel Every step in a funnel must come from the same Fact Table. Changing the Fact Table on the first step rewrites it for all steps and clears each step's row filters, so pick the table before you write your filters. The ability to choose a different Fact Table per step is coming soon! ::: ### Step settings Each step has the following settings: - **Step name**: The label shown in the results table, the funnel chart, and CSV exports. Defaults to `Step N`. - **Row filters**: What counts as completing this step, such as `event_name = add_to_cart`. These work the same as [Row Filters](/app/metrics#row-filters) on any other metric, including saved filters and custom SQL expressions. - **Optional step**: Units who skip this step can still complete later steps. See [Optional steps](#optional-steps). - **Conversion window**: How long the unit has to reach this step. Enter a number and a unit of `minutes`, `hours`, `days`, or `weeks`. ### Metric-level settings Funnel metrics support [Metric Windows](/app/metrics#metric-windows), a [metric delay](/app/metrics#metric-delay), a target MDE, and [Bayesian priors](/app/metrics#bayesian-priors). The metric window bounds which events are eligible at all; step conversion windows then apply inside that envelope. [Metric Slices](/app/metrics#metric-slices) are not supported for funnel metrics. ## How steps are resolved Funnels are sequential: steps must happen in order, and a unit can complete a later step without having completed an optional one along the way. For each step, GrowthBook takes the **earliest** qualifying event that falls inside that step's window. The window is anchored on the nearest prior **required** step. The first step is anchored on experiment exposure, and so is any step whose prior steps are all optional. Take a three-step funnel where a unit is exposed at 10:00: | Step | Conversion window | Anchored on | Counts if the event happens | | ----------------- | ----------------- | ----------------- | ---------------------------------------------- | | 1. Viewed product | 1 hour | Exposure at 10:00 | Between 10:00 and 11:00. Say it lands at 10:05 | | 2. Added to cart | 2 hours | Step 1 at 10:05 | Between 10:05 and 12:05. Say it lands at 11:30 | | 3. Purchased | 1 day | Step 2 at 11:30 | Between 11:30 and 11:30 the next day | A step with no conversion window has no upper bound. It counts as long as it happens after the previous non-optional step and inside the metric window. ### Optional steps An optional step is measured like any other step: you still get its conversion rate, its lift, and its own row in the results. What changes is that it does not anchor or filter the steps after it. In the example above, if **Added to cart** were optional and the unit never added to cart, **Purchased** would be measured within 1 day of **Viewed product** instead. Units who did add to cart are still anchored on their view, not their add to cart. Use optional steps for parts of a journey you want to measure but that are not required to continue. :::note Partial data Metric level conversion windows are a hard cap on the completion of all steps. Data that lands after a conversion window has closed are not considered. If the experiment has **Skip Partial Data** enabled, where it controls which recent exposures are excluded as not having had time to convert, time to convert is measured as the sum of each steps conversion windows, capped by the metric window. ::: ## Every step shares one denominator This is the most important thing to know about funnel metrics: **each step is measured against all exposed units, not just the units who completed the previous step.** If 1,000 units are exposed to a variation and 400 view a product, 120 add to cart, and 60 purchase, the three steps report 40%, 12%, and 6%. They do not report 40%, 30%, and 50%. That choice is critical for deriving the correct interpretation from your results. Using the conversion total from the previous step as the denominator for your next step will mislead most experimenters, because it carries forward the indirect effect of the last step into the current step's analysis. **Example: why step-to-step conversion rates can be misleading** Suppose you run an experiment on a two-step funnel: Add to Cart -> Purchase, with 1,000 users in each variation. Imagine we know the true effect: the treatment brings in many new, lower-intent cart-adders, but checkout rates are unchanged by this experiment (low-intent users still checkout at the same rates as before and high-intent users still checkout at the same rates as before. | Metric | Control | Treatment | | ------------------------------------------ | --------- | --------- | | Users assigned | 1,000 | 1,000 | | Added to cart | 200 (20%) | 400 (40%) | | Purchased | 100 | 140 | | Cart → Purchase rate (conditional) | 50% | 35% | | Assignment → Purchase rate (unconditional) | 10% | 14% | The step-to-step, conditional conversion rate (Cart → Purchase) fell from 50% to 35% so it looks like the treatment hurt checkout. But nothing about checkout changed. The control arm's carts had 200 high-intent shoppers; the treatment arm added 200 casual browsers who convert at ~20%, dragging the average down. Reading the 50% -> 35% drop as "checkout got worse" conflates the treatment's effect on who reaches the cart with its effect on checkout itself. The last row is what GrowthBook would represent as the second step conversion rate and it tells the true story in this case: purchases per assigned user rose from 10% to 14%. That comparison is safe because randomization guarantees the two groups of 1,000 are comparable, comparing the groups of cart-adders are not. This is an example of the well known "post-treatment bias" problem, and is akin to what happens when sample ratio mismatch is a sign of assignment bias. Step-to-step conversion rates are still available in the [funnel chart](/app/experiment-results#funnel-metric-results) in the metric drilldown, but we do not currently provide them as a top-level metric with the confidence intervals and statistics used for decision making. Those are reserved for the safer, unconditional conversion rates. The metric's own value is the share of exposed units who reach the **last** step, which is your end-to-end completion rate. ## Where funnel metrics can be used Funnel metrics work as **goal**, **secondary**, and **guardrail** metrics on experiments and reports, including with dimensions, date results, and [Experiment Dashboards](/app/experiment-dashboards). The following are not supported yet: | Feature | Behavior | | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Activation metrics | Funnel metrics are hidden from the activation metric dropdown. Use a proportion metric instead. | | [Metric Slices](/app/metrics#metric-slices) | Not supported. The drilldown's Slices tab is replaced by a Funnel tab. | | [CUPED and regression adjustment](/statistics/cuped) | Always off for a funnel and its steps, because the funnel query does not emit covariate columns. CUPED support for funnel metrics is coming soon! | | Standalone metric analysis | The metric page cannot chart a funnel on its own. It is analyzed in the context of an experiment. | | [Product Analytics Explorer](/app/product-analytics/explorer) | Funnel metrics cannot be charted in the Explorer or in a metric explorer dashboard block, and this is coming soon! In the meantime, use the [Funnel explorer](/app/product-analytics/explorer#funnels) for ad-hoc funnel analysis. | | Incremental Refresh | Funnel metrics are not supported for Incremental Pipeline experiments yet, but support is coming soon! For now, remove them from the experiment, or run the experiment without incremental refresh. | | Capping and quantile settings | Not supported. | ### Supported data sources Funnel metrics run on Postgres, ClickHouse (including the [Managed Warehouse](/app/managed-warehouse)), BigQuery, Snowflake, Athena, Presto, Databricks, and Redshift. :::warning MySQL, MSSQL, and Vertica cannot run funnel queries. Adding a funnel metric to an experiment on one of these data sources fails the query with an unsupported data source error naming the funnel metrics involved. Mixpanel and Google Analytics do not support Fact Tables at all. ::: ## Funnel metrics in experiment results In the results table, a funnel metric is one row you can expand into a row per step. Each step row shows that step's conversion rate, lift, and significance against the shared denominator, and steps always stay in funnel order no matter how you sort. Clicking the metric opens the drilldown with a **Funnel** tab that charts each variation's funnel side by side. See [Funnel Metric Results](/app/experiment-results#funnel-metric-results) for the full walkthrough. ## Managing funnel metrics with the API Create a funnel metric by setting `metricType` to `funnel` and passing `funnelSettings`. Omit `numerator` and `denominator`; they are not allowed on funnel metrics. The metric inherits its data source from the first step's Fact Table. ```bash curl -X POST 'https://api.growthbook.io/api/v1/fact-metrics' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "name": "Checkout Funnel", "owner": "you@example.com", "metricType": "funnel", "funnelSettings": { "steps": [ { "name": "Viewed product", "factTableId": "ft_events", "rowFilters": [ { "column": "event_name", "operator": "=", "values": ["view_product"] } ], "optional": false, "conversionWindow": { "unit": "hours", "value": 1 } }, { "name": "Added to cart", "factTableId": "ft_events", "rowFilters": [ { "column": "event_name", "operator": "=", "values": ["add_to_cart"] } ], "optional": true, "conversionWindow": { "unit": "hours", "value": 2 } }, { "name": "Purchased", "factTableId": "ft_events", "rowFilters": [ { "column": "event_name", "operator": "=", "values": ["purchase"] } ], "optional": false, "conversionWindow": { "unit": "days", "value": 1 } } ], "ordering": "sequential", "concurrencyWindowSeconds": 0 } }' ``` Field reference for `funnelSettings`: - **steps**: Required. An ordered array of 2 to 20 steps. Each step needs `name`, `factTableId`, `rowFilters`, and `optional`; `conversionWindow` is optional and may be `null`. - **ordering**: Optional, defaults to `sequential`. Only `sequential` is accepted today. - **concurrencyWindowSeconds**: Optional, defaults to `0`. Tolerance in seconds for events that arrive slightly out of order, applied to the lower bound of each step's window. `GET` responses for funnel metrics include `funnelSettings` and omit `numerator`. Listing fact metrics filtered by `factTableId` returns funnel metrics whose steps use that Fact Table. Per-step results are exposed in the [bulk experiment results API](/app/experiment-bulk-results-api) as derived metric ids of the form `?step=`, zero-indexed, named `Funnel name: Step name`. These ids are read-only. They cannot be created, queried, or added to an experiment on their own. ## What's next - [Understanding Experiment Results](/app/experiment-results#funnel-metric-results) for reading funnel results in an experiment. - [Metrics and Fact Tables](/app/metrics) for row filters, metric windows, and the other metric types. - [Funnel explorer](/app/product-analytics/explorer#funnels) for ad-hoc funnel analysis outside of an experiment. --- ## Legacy Metrics import Anchor from '@site/src/components/Anchor' # Legacy Metrics Metrics are what your experiments are trying to improve (or at least not hurt). GrowthBook has a very flexible and powerful way to define metrics. :::warning Legacy Metrics This page describes Legacy Metrics. We highly recommend using **Fact Tables** to define metrics instead. [Check out the docs](/app/metrics) ::: ## Conversion Types Metrics can have different units and statistical distributions. Below are the ones GrowthBook supports: | Conversion Type | Description | Example | Default aggregation (SQL) | | --------------- | ------------------------------- | ---------------- | ------------------------- | | binomial | A simple yes/no conversion | Created Account | 1/0 per unit | | count | Sums conversion values per user | Pages per Visit | `SUM(value)` per unit | | duration | How much time something takes | Time on Site | `SUM(value)` per unit | | revenue | The revenue gained/lost | Revenue per User | `SUM(value)` per unit | For experiment analysis, each of these metric types uses some aggregation (often defaulting to `SUM`) per user and then takes an average with respect to the total number of users. In the case of SQL metrics, the only meaningful difference between count, duration, and revenue is how we render the units in the metric and experiment results pages. :::info USD by default Revenue Metrics are displayed in USD by default. You can change your display currency under **Settings** → **General** → **Metric Settings** ::: ## Query settings For metrics to work, you need to tell GrowthBook how to query the data from your data source. There are a few ways to do this depending on your data source. ### 1. SQL (recommended) If your data source supports SQL, this is the preferred way to define metrics. You can use joins, subselects, or anything else supported by your SQL dialect. Your SELECT statement should return one row per "conversion event". This may be a page view, a purchase, a session, or something else. The end result should look like this: | user_id | timestamp | value | | ------- | ------------------- | ----- | | 123 | 2021-08-23 12:45:14 | 10 | | 456 | 2021-08-23 12:45:15 | 5.25 | Metrics can support one or more types of identifiers. The above example assumes the metric only supports a single id type called `user_id`, but you would add additional columns if you need to support other ones. #### Non-binomial metrics For count, revenue, and duration metrics metric types, the value represents the count, duration, or revenue from that single conversion event. In the case of multiple rows for a single user, the values will be summed together or we will use a custom aggregation that you can specify. Therefore a `count` metric can really be any arbitrary metric whose `value` you want to sum at the user level before taking an average per variation. If you use Segment to populate your data warehouse, the SQL for a `Revenue per User` metric might look like this: ```sql SELECT -- Assuming you support 2 identifier types - 'user_id' and 'anonymous_id' user_id as user_id, anon_id as anonymous_id, received_at as timestamp, grand_total as value FROM purchases ``` If you wanted to count the number of conversion rows per user, you can simply set `1 as value` in your SQL query and then the default SUM aggregation will count the number of rows per user. #### Binomial metrics Binomial metrics don't need a `value` column (the existence of a row means the user converted). You would only need to return the following columns, representing users and when they "converted" on this binomial metric: | user_id | timestamp | | ------- | ------------------- | | 789 | 2022-08-23 12:45:14 | | 111 | 2022-08-23 12:45:15 | When we go to conduct experiment analysis, any `user_id` that has a conversion in the appropriate time window will be counted as a `1`, while all other users will be counted as a `0`. Then we can compute the proportion of users in an experiment variation who converted. #### SQL Templates We use \{\{[Handlebars](https://handlebarsjs.com/guide/#language-features)\}\} to compile the sql into what is actually called to your database. This allows you to create template metrics that can be copied and reused by changing the variable values associate with the metrics. You can use the following user configurable variables within SQL templates: - **eventName** - The event name associated with this metric. This can then be referenced in your sql template as `{{eventName}}`. Depending upon how your data is structured you can then incorporate it as part of the table name, if each event has its own table, or as part of a where clause limiting the rows returned to where a certain column equals the eventName. - **valueColumn** - The column in your datawarehouse table with the metric data. This can then be referenced in your sql template as `{{valueColumn}}`. For example you might have `{{valueColumn}} as value` to extract out the value from the table. Read more about [SQL Templates](/app/sql-templates) and all the different variables that are available. #### Denominator (Ratio / Funnel Metrics) By default, metrics are evaluated against all users in an experiment: `(# users who converted) / (# users in experiment)` You can instead choose another metric to use as the denominator. ##### Funnel Metrics When the denominator is a simple binomial (conversion) metric, then it acts just like an "activation metric" in an experiment. It filters the users who are included in the analysis to those who first convert on this denominator metric. For example, if you want to look at what percent of users checkout after viewing a cart, it can be described as `% checkout / % viewed cart`. This requires creating two metrics: 1. `Viewed Cart` - selects all users who viewed a cart 2. `Viewed Cart -> Checkout` - selects all users who checked out and picks `Viewed Cart` as the denominator. :::tip Fact Tables have a dedicated [Funnel Metric](/app/metrics/funnel) type for this. One funnel metric defines both steps, reports a conversion rate for each of them, and does not require chaining two metrics together. ::: ##### Ratio Metrics When the denominator is a count metric, then things are a little different. Instead of acting like a filter, we calculate both metrics and treat the value as a ratio. - The mean is `sum(metric) / sum(denominator)` - The standard deviation is calculated using the **Delta method** For example, if you want to look at the Average Order Value (AOV), what you're really looking for is `total revenue / number of orders`. This also requires creating two metrics: 1. `Orders per User` - selects the count of orders for each user 2. `AOV` - selects total revenue per user and picks `Orders per User` as the denominator. ##### Quantile Metrics [Quantile Metrics](/statistics/quantile) are available only as Fact Tables, as described [here](/app/metrics#quantile-metrics). ### 2. Javascript (Mixpanel only) We query Mixpanel data sources using their proprietary JQL language based on Javascript. This allows for extreme flexibility when defining metrics. All metrics at minimum need to specify an **Event Name** which must exactly match what is used in Mixpanel. You can use `OR` to match against multiple events. For example `viewed_cart OR purchased` You can optionally add **Conditions** which filters the events further based on properties. For example, if Event Name is `Page view`, you can add a condition `path = "/blog"`. Count, duration, and revenue metrics have two additional steps. We first extract all event values for a user into an array and then reduce that array down to a single number, which is the final metric value for the user. #### Conditions Conditions in Mixpanel are very powerful. They consist of a **Property**, an **Operator**, and a **Value**. Multiple conditions are joined with an AND. The **Property** can either be the name of an event property or a javascript expression. Some examples: - `amount`, equivalent to `event.properties.amount` - `event.time` - `event.properties.city + ", " + event.properties.country` The **Operator** is one of the following: - equals - does not equal - is greater than - is greater than or equal to - is less than - is less than or equal to - matches the regex - does not match the regex - custom javascript The `custom javascript` operator is special. The **Value** is a javascript expression that evaluates to either `true` or `false` (access the property value with the `value` variable). It lets you do arbitrarily complex filtering. For example: ```js value 10; ``` For all other operators, the condition should read like an English sentence. For example: ```js // property operator value country equals US // Will render as event.properties.country == "US" ``` #### Event Value For count, revenue, and duration metrics, we need to know what the "value" of the event is. The **Event Value** is a Javascript expression to extract a value from a raw Mixpanel event. If you are just extracting a single property as-is, you can just enter the property name as a shortcut. Otherwise, you can reference the `event` variable in your expression. Here are some example Event Value expressions: - `grand_total`, equivalent to `event.properties.grand_total` - `1` (hard-code the value to a specific number) - `(event.properties.endTime - event.properties.startTime) / (60 * 60)` (difference in hours between two unix timestamps) - `new Date(event.time).toISOString().substr(0, 10)` (event timestamp in YYYY-MM-DD format) For count metrics, you can leave Event Value blank and it will default to hard-coding the value to `1`, which is perfect for when you just want to count the number of events and don't care about specific properties. #### User Value Aggregation For count, revenue, and duration metrics, we need to know how to aggregate the event values together, in case a single user has multiple matching events. The **User Value Aggregation** is another Javascript expression that reduces an array of Event Values to a single number (or null if the user did not convert). Reference the variable `values` in your expression. There are a few built-in helper functions: - `count(values)` - `countDistinct(values)` - `sum(values)` - `min(values)` - `max(values)` - `avg(values)` - `median(values)` - `percentile(values, p)` (p is a number between `0` and `100`) You can use your own custom expression too if you want. For example, this is the equivalent of `sum(values)`: ```js values.reduce((sum, n) => sum + n, 0); ``` If the aggregation is left blank, we do `sum(values)` by default. ### 3. SQL Query Builder (legacy) The query builder prompts you for things such as table/column names and constructs a SQL query behind the scenes. We only recommend this for extremely simple metrics. Inputting raw SQL is far more flexible. ## Behavior The behavior tab lets you tweak how the metric is used in experiments. Depending on the metric type and datasource you chose, some or all of the following will be available: ### What is the Goal? For the vast majority of metrics, the goal is to increase the value. But for some metrics like "Bounce Rate" and "Page Load Time", lower is actually better. Setting this to "decrease" basically inverts the "Chance to Beat Control" value in experiment results so that "beating" the control means decreasing the value. This will also reverse the red and green coloring on graphs. ### Capped Value Large outliers can have an outsized effect on experiment results. For example, if your normal revenue per user \$40 and someone happens to make a \$5000 order, whatever variation that person is in will be much more likely "win" any experiment because that one order is an outlier. Capping (also known as winsorization) works by ensuring that all aggregate unit (e.g. user) values are no more than some value. So in the above example, if the cap was \$100, the \$5000 purchase will still be counted, but the aggregated value for that user will be capped at \$100 and will have a much smaller effect on the results. It will still give a boost to whatever variation the person is in, but it won't completely dominate all of the other orders and is unlikely to make a winner just on its own. Another way to think about this is that you are slightly biasing your results by truncating large values, but you are reducing variance to prevent the outsized effect of outliers. There are two ways to cap metric values in GrowthBook: **1. Absolute capping** - if set above zero, all aggregated user values will be capped at exactly this value. For example, if the cap is \$100 on total revenue per user, then after we sum all of a users orders up, any user with an aggregate sum of greater than \$100 will be set to \$100. **2. Percentile capping** - when this is set to between 0 and 1, it uses that percentile to select a cap based on the data in your experiment so far. This cap is therefore specific to each experiment and specific to each analysis run in that experiment if new data has come in. It works like so: after we calculate the unit-level aggregate values for all units (e.g. users) during an experiment analysis, we find the specified percentile of these unit-level aggregates and then cap these aggregated values at this percentile. Using the above example, if you were to specify percentile capping with a value of `0.95`, then we find the 95th percentile of total revenue per users (say this turns out to be \$135). We then cap those user-level aggregates at \$135. You can additionally choose to ignore zeros, which will compute the percentile without including any user aggregated zero values. This is useful if you have a lot of zero values and you don't want to have to fine tune the percentile to avoid setting the cap too low. Because the percentile cap depends on the data in your experiment, it can be different from experiment to experiment, or even analysis to analysis. To find out what value was actually used for capping you can do the following: on the Experiment Results tab, click the three dot menu in the top right and select "View Queries". Each percentile capped metric will have a column with the `main_cap_value` that was used to cap that metric and represents the computed percentile of unit-level aggregate values. ### Metric Windows When used in an experiment, we only consider rows of a Metric where the timestamp is greater than or equal to the first time the user was exposed to the experiment. In other words, if someone purchases something before seeing your experiment, it won't be included in the analysis. This behavior is ideal for the vast majority of metrics, but you can change it with the Metric Delay setting if desired (see below). There are three window settings one can use to configure the metric date window. Each of them defines the lower and upper date range of the metric to use for each user: - **None** (default) - Lower bound: user's first exposure plus the metric delay - Upper bound: experiment end date - **Conversion Window** - Lower bound: user's first exposure plus the metric delay - Upper bound: the lower bound + the length of the conversion window - **Lookback Window** - Lower bound: the experiment end date minus the lookback window OR the user's first exposure plus the metric delay, whichever is later - Upper bound: experiment end date Here's a graphical representation of these three window types for a hypothetical User 1: ![Metric Windows](/images/metric-windows.png) Here's a second example for a hypothetical User 2, who joins the experiment late. Notice that the conversion window can extend beyond the experiment end date. ![Metric Windows (User 2)](/images/metric-windows-user-2.png) Why might you choose one window over another? **None** - The simplest. Use all data available. This is useful for using as much data associated with users in your experiment and will combine any behavior that is right after experiment exposure as well as long run behavior within the experiment time frame. **Conversion window** - Conversion windows allow you to only look at events that are tied to the first exposure to an experiment. This can help if, for example, you are tracking purchases and you only want to measure the effect of an experiment in a checkout flow on purchases made soon after seeing that checkout flow. Using a conversion window can reduce the noise from user behavior not related to an experiment. However, if you set the window too short, you may not capture users that return a few days later and were influenced by the experiment. **Lookback window** - Lookback windows are good for capturing long run impacts of an experiment on regular behavior like user log ins or page views. They have two main advantages: (1) You can mitigate the novelty effect of an experiment; if you are testing a new recommendation algorithm, at first users may react a certain way to the experiment, but eventually they may adjust and so may their behavior. In these cases, you may just want to look at the last 14 days of an experiment. (2) Lookback windows help you focus on the long run effects of an experiment. Much of experimentation is about building a better product; by focusing on impact of an experiment after it has been live for a week or two, you may get a better picture of the long run impact of launching the experiment. Larger companies who measure long run logged in behavior want to ensure their experiments have lasting effects and often rely on lookback windows to make shipping decisions However, lookback windows may not be right if you are testing a feature on logged-out or anonymous users and are measuring simple purchase conversions or something similar. In these cases, you might end up with many logged-out users who, in the long run, simply have no metric data associated with them. ### Metric Delay Conversions within the first X hours of being put into an experiment are ignored (default = `0`). This is useful for metrics like "day 2 retention". In that case, if your underlying table reports whether a user is retained on any given day, you could set a metric delay to `24` hours. #### Negative metric delays The metric delay can also be negative to include some conversions **before** a user is put into an experiment. For example, a value of `-2` would mean conversions up to 2 hours before will be included. You might be wondering when this would ever be useful. Imagine the average person stays on your site for 60 seconds and your experiment can trigger at any time. If you just look at the average time spent after the experiment, the numbers will lose a lot of meaning. A value of `20 seconds` might be horrible if it happened to someone after only 5 seconds on your site since they are staying a lot less time than average. But, that same `20 seconds` might be great if it happened to someone after 55 seconds since their visit is a lot longer than usual. Over time, these things will average out and you can eventually see patterns, but you need an enormous amount of data to get to that point. If you set the metric delay to something negative, say `-0.5` (30 minutes), you can reduce the amount of data you need to see patterns. For example, you may see your average go from 60 seconds to 65 seconds. Keep in mind, these two things are answering slightly different questions. `How much longer do people stay after viewing the experiment?` vs `How much longer is an average session that includes the experiment?`. The first question is more direct and often a more strict test of your hypothesis, but it may not be worth the extra running time. ### Bayesian Priors Your organization can set default priors for Bayesian analyses that are used by all metrics. However, you can also set metric specific priors by opening the Edit Metric modal from the Metric page, clicking on Advanced Settings, and turning on the metric override. This will allow you to set a custom prior for that metric. Additionally, you can use experiment metric overrides to further customize these priors for each experiment. You can read more about Bayesian priors on [our statistical details page](/statistics/details). ## Auto Generate Metrics When using GrowthBook with certain event trackers, we may be able to generate metrics for you automatically by identifying the unique events tracked by your event tracker. This is currently only supported for a few event trackers (listed below), but we are working to expand this list. If you are using one of the supported event trackers and would like to see what metrics we can create for you, head to the `Metrics` page in GrowthBook and select the `Discover Metrics` button. :::info Auto Generate Metrics limitations When querying your datasource to identify unique events, we're currently only looking at events in the last 7 days. ::: ### Supported Event Trackers - Segment - Rudderstack - Google Analytics 4 (GA4) - Amplitude ## Examples Let's walk through some examples of creating binomial, count, and retention metrics with GrowthBook. For all of the metrics below, let's pretend we have some table called `events`, which has one row per event tracked to your warehouse. For each row we have the following columns: - `user_id` - the id of the user - `timestamp` - the time the event was counted - `event_name` - the name of the event, we'll focus on 'purchase' as a key event types - `value` - the total value of the event, in this case the total value of a purchase From this table, we can build many different metrics: | Name | Metric Type | SQL | Aggregation | Denominator | Metric Delay | Metric Window | | -------------------- | --------------- | ------------------------------------------------------------------------------- | ------------- | ------------------- | --------------- | ----------------- | | Any Purchase | Binomial | SELECT user_id, timestamp FROM events WHERE event_name = 'purchase' | n/a | | 0 | | | Number of Purchases | Count | SELECT user_id, timestamp, 1 as value FROM events WHERE event_name = 'purchase' | default (SUM) | | 0 | | | Order Value | Revenue | SELECT user_id, timestamp, value FROM events WHERE event_name = 'purchase' | default (SUM) | | 0 | | | Average Order Value | Revenue (ratio) | SELECT user_id, timestamp, value FROM events WHERE event_name = 'purchase' | default (SUM) | Number of Purchases | 0 | | | 7-Day Retention | Binomial | SELECT user_id, timestamp FROM events | n/a | | 24\*7=144 hours | | | Active User Last 14d | Binomial | SELECT user_id, timestamp FROM events | n/a | | 0 | Lookback - 14 day | If you wanted to only count purchases and purchase values made in the 72 hours after a user's first exposure to an experiment, then you could set the metric window to a conversion window of 72 hours. If you wanted to just count any user who made a purchase any time after experiment exposure, then set the metric window to none. ## Migrating Legacy Metrics to Fact Tables We recommend using Fact Tables for all new metrics. Legacy metrics will continue to be supported, but they will not get any new features. If you would like to migrate existing legacy metrics to Fact Tables, there are a few differences to be aware of. ### Reusable Definitions Metric definitions have been split up into a few different reusable pieces. It's a couple more steps to create your first metric, but it should drastically simplify adding subsequent ones and building out your metric library. - The SQL and supported user identifiers are defined in the **Fact Table** - The display formatting (e.g. currency vs duration) is attached to a Fact Table **Column** - Any WHERE clauses for metrics are defined as **Filters** (e.g. `device_type = 'mobile'`) Once these pieces are in place, defining Metrics is now much easier. The form went from 13+ steps down to just 4 and no longer requires any specialized SQL or database knowledge. ### Metric SQL Before, SQL for a metric would select at most 1 numeric column and it had to be named `value`. With Fact Tables, you can have as many numeric columns as you want and there are no naming restrictions. This allows you to have 1 complex SQL definition be re-used across many related metrics. Also, Custom aggregations are no longer supported. Fact Tables only support a few pre-defined aggregations - `COUNT` and `SUM` (we may add more in the future). Using these pre-defined aggregations greatly simplifies the queries we run and enables advanced performance and cost optimizations. ### Metric Types Metric types have changed. - Binomial Metrics have been renamed to **Proportion Metrics** - Count, Duration, and Revenue are all now just **Mean Metrics** (the display formatting is controlled by the Fact Table columns now) - Instead of just adding a denominator to any metric, there is now a dedicated type for **Ratio Metrics**. ### Ratio Metrics There are a lot of changes to how Ratio Metrics are defined and how they behave. Ratio metrics are now self-contained. Previously, the denominator would just be a pointer to an entirely separate metric. This allowed some weird edge cases like the denominator itself being a ratio metric with it's own denominator. With Fact Tables, you define both the numerator and denominator in one flow and these nested denominator edge cases are no longer supported. During analysis, Ratio metrics used to behave like a Funnel - the denominator had to happen first before the numerator. This was unintuitive for many people, so With Fact Tables, we changed them to act like true ratios - we calculate the numerator and denominator independently and then divide them. To measure an ordered sequence of events, use a [Funnel Metric](/app/metrics/funnel) instead. Lastly, Ratio metrics now have a single metric window and capping behavior. Previously, this would be controlled separately for the numerator and denominator. Because of this change, only "percentile" capping is allowed for ratio metrics. --- ## Metric Examples and Use Cases import AddMetricButton from '@site/src/components/AddMetricButton' import HostSelector from '@site/src/components/HostSelector' # Metric Examples and Use Cases Fact Tables in GrowthBook are a powerful way to quickly build out a library of metrics. Below are some common types of tables in your data warehouse and how to map them to Fact Tables with example SQL and metric definitions. The **Add to GrowthBook** buttons will open GrowthBook and let you customize the metric before adding it to your account. ## Raw Events Table The most common form of analytics data is a raw event stream. Every time a user does something, you append a new row to an events table in your warehouse. All types of events from all users are intermixed in this one table. Fact Tables in GrowthBook work really well with raw events. When defining your Fact Table in GrowthBook, there are 4 important types of columns you will want to select: 1. Timestamp - When the event occurred 2. Ids - Who triggered the event (user id, device id, etc.) 3. Event Type - What action was performed (page view, start checkout, add friend, etc.) 4. Context - Any helpful user or app context (geo-location, current url, account plan, etc.) When creating metrics, you will almost always want to filter on the Event Type. To make this easier, you can edit the column representing Event Type and check the “Prompt all metrics” box (after creating your Fact Table). This helps GrowthBook better streamline the UI when creating metrics. ![Event Type Filter](/images/metrics/inline-filter.png) ### Example SQL: Google Analytics 4 (GA4) Events When you configure Google Analytics 4 to sync events into BigQuery, they get written to a raw events table (actually multiple tables partitioned by day, but that’s more of an implementation detail) Here’s example SQL for a fact table. Comments added to highlight the 5 column types above ```sql SELECT -- Timestamp TIMESTAMP_MICROS(event_timestamp) as timestamp, -- Ids user_id, user_pseudo_id as anonymous_id, -- Event type event_name, -- Context event_value, -- e.g. the page view path geo.country, device.category as device_category FROM `events_*` ``` For faster and cheaper queries, we can take advantage of the date partitioning table scheme. GrowthBook lets you use date template variables that are evaluated right before executing the SQL. By doing this, we can restrict our query to just the relevant dates for the experiment we are analyzing. Change the end of the query and add the following WHERE clause: ```sql FROM `events_*` WHERE -- Daily tables ((_TABLE_SUFFIX BETWEEN '{{date startDateISO "yyyyMMdd"}}' AND '{{date endDateISO "yyyyMMdd"}}') -- Realtime streaming intraday table OR (_TABLE_SUFFIX BETWEEN 'intraday_{{date startDateISO "yyyyMMdd"}}' AND 'intraday_{{date endDateISO "yyyyMMdd"}}')) ``` ### Example Metrics #### Percent of users who submit a form 1. Select `Proportion` as the metric type 2. Choose `form_submit` as the event type #### Percent of users who visit the checkout page 1. Select `Proportion` as the metric type 2. Choose `page_view` as the event type 3. Add a “Row Filter” and select `page_path = '/checkout'` #### Page Views per User 1. Select `Mean` as the metric type 2. Choose `page_view` as the event type 3. Select `Count of Rows` as the value #### Unique Page Views per User 1. Select `Mean` as the metric type 2. Choose `page_view` as the event type 3. Select `event_value` as the column 4. Select `COUNT DISTINCT` as the aggregation #### Active Days per User 1. Select `Mean` as the metric type 2. Choose `Distinct Dates` as the column #### Checkout completion rate 1. Select `Ratio` as the metric type 2. For the **numerator** (confirmation) 1. Choose `page_view` as the event type 2. Add a Row Filter for `page_path = '/confirmation'` 3. Select `Unique Users` as the value 3. For the **denominator** (checkout) 1. Choose `page_view` as the event_type 2. Add a Row Filter for `page_path = '/checkout'` 3. Select `Unique Users` as the value ## Modeled Tables Another common way to store data in a warehouse is with modelling. This usually involves a tool like `dbt` that transforms raw event logs into multiple well-structured tables. There are also some event tracking tools like Segment which do this automatically. For our purposes, we are only considering modeled tables that do not include a GROUP BY in the data pipeline - in other words, the rows are still at the raw event-level. Check out the Pre-aggregated Tables section below if this is not the case. In GrowthBook, we recommend creating a separate Fact Table for each one of your modeled tables. At the very least, you will need a Timestamp and IDs of who performed the action. Any additional columns you add are optional and can be used to build out metrics. ### Example SQL: E-Commerce Orders Table ```sql SELECT -- Timestamp received_at as timestamp, -- IDs user_id, -- Other custom columns num_items, grand_total, shipping_type FROM orders ``` Because this is a modeled table just for orders, we can select event-specific columns like `num_items`, `grand_total`, and `shipping_type`. These columns will help us build out a rich set of metrics. This is one of the big benefits over using a raw events table directly where the columns you select need to be more generic and work across a wide range of different events. Similar to raw event tables, you can optionally add a WHERE clause on dates for faster/cheaper queries: ```sql FROM orders WHERE timestamp BETWEEN '{{startDate}}' AND '{{endDate}}' ``` ### Example Metrics #### Users who Purchased 1. Select `Proportion` as the metric type and that’s it! #### Users who Purchased Multiple Times 1. Select `Proportion` as the metric type 2. Add a “User Filter” based on `Count of Rows` 3. Enter `>= 2` as the User Filter value = 2" } }}/> #### Orders per User 1. Select `Mean` as the metric type 2. Select `Count of Rows` as the metric value #### Revenue per User 1. Select `Mean` as the metric type 2. Select `SUM(grand_total)` as the metric value #### Average Items per Order 1. Select `Ratio` as the metric type 2. For the numerator, select `SUM(num_items)` 3. For the denominator, select `Count of Rows` #### Users Without Orders 1. Select `Proportion` as the metric type 2. Add a “User Filter” for `Count of Rows = 0` 3. Under Advanced Settings -> Display Settings, change the goal to `Decrease the Metric Value` #### Percent of Orders with Free Shipping 1. Select `Ratio` as the metric type 2. For the numerator, add a row filter for `shipping_type = 'free'` 3. Select `Count of Rows` for both the numerator and denominator ## Pre-Aggregated Tables Pre-aggregated tables include a GROUP BY in the data pipeline to compress raw event-level data down to fewer rows. This is usually done when querying raw data directly is prohibitively expensive. GrowthBook supports pre-aggregated tables as long as they satisfy two requirements: - Must be grouped by both user and date - Pre-aggregated columns can only be basic sums or counts. No averages, percentiles, count distinct, or complex derived formulas that break statistical assumptions. Using pre-aggregated tables for experimentation comes with additional complexities and downsides, so we highly recommend sticking with event-level data whenever possible. ### Example SQL: Social Media Interactions (User/Date Rollup) If you have a social media app, the number of raw interactions might be huge. A user/date rollup table can drastically reduce the size of the data set. Here is an example Fact Table SQL definition in GrowthBook that references a user/date rollup table. ```sql SELECT date as timestamp, user_id, -- Aggregated columns (sums/counts only) interactions, likes, comments FROM user_date_rollup ``` Pre-aggregated tables have a problem. The `timestamp` column isn’t very accurate. All events for a day are recorded as happening at the beginning of the day (usually at midnight). We only consider Fact Table rows with a timestamp after a user first enters an experiment. That basically means a user’s first day of data will be excluded from the analysis. This is a big deal because those first few hours after entering a test are often the most important. We can fix this by incrementing all timestamps by 1 day in our Fact Table definition (specific SQL varies by db engine): ```sql SELECT DATE_ADD(date, INTERVAL '1 day') as timestamp, user_id ... ``` This introduces a new problem however - we will now potentially include actions from before a user saw an experiment, which can introduce noise to the results. In our experience, this is usually the lesser of two evils and thus, this is the approach we recommend. Of course, you can solve both of these problems by not pre-aggregating in the first place, but we understand sometimes that’s just not an option. ### Example Metrics #### Percent of Users with High Engagement 1. Select `Proportion` as the metric type 2. Add a “User Filter” and select `SUM(interactions)` as the value to filter on 3. Add some threshold as the User Filter value. Something like `> 100` 100" } }}/> #### Interactions per User 1. Select `Mean` as the metric type 2. Select `SUM(interactions)` as the value #### P90 Likes per User 1. Select `Quantile` as the metric type 2. Check the "Group by user" box 3. Select `SUM(likes)` as the per-user value 4. Select `P90` as the quantile level #### Comment-to-Like Ratio 1. Select `Ratio` as the metric type 2. Select `SUM(comments)` as the numerator value 3. Select `SUM(likes)` as the denominator value --- ## Metrics and Fact Tables import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; # Metrics and Fact Tables Metrics are how you measure success and failure for your business. An e-commerce company might have metrics related to conversion rates, revenue, and cart abandonment. A B2B SaaS company might have metrics around subscription trials, cancellations, and usage. Every company is different and GrowthBook is flexible enough to support a wide variety of use cases. The primary way to create metrics in GrowthBook is with **Fact Tables** where you write SQL to pull a set of rows and then define multiple metrics on top of those rows. There is also a [legacy method](/app/metrics/legacy) where each metric has its own self-contained SQL. Fact Tables are easier to use, faster to query, and more powerful, so we highly recommend using them for all new metrics. If you learn best by example, you can jump straight into [Metric Examples and Use Cases](/app/metrics/examples). ## Fact Tables Fact Tables are defined by a SQL SELECT statement and are the base that metrics are built on top of. At a minimum, a Fact Table must select the following columns: - `timestamp` - The date the event happened - One column per supported identifier type (e.g. `user_id` and `anonymous_id`). The possible identifier types depend on your data source settings. Any other columns you return can be used to create metrics or used as a filter. Here's a full example: ```sql SELECT -- Required columns purchase_date as timestamp, user_id, anonymous_id, -- Additional columns (depends on your use case) amount, coupon_code, device_type FROM orders WHERE status = 'completed' ``` This is an example of a "modeled table", which represents a specific order event and has dedicated columns related to orders. GrowthBook also works great with raw event stream tables and even some pre-aggregated tables. Check out [Metric Examples and Use Cases](/app/metrics/examples) for more info and examples of these different data schemas and table types. ### Columns When you create a Fact Table, we query the first few rows and inspect the returned data to determine which columns you've selected and what data types they are. This process isn't perfect. For example, if the first few rows happen to have `null` as the value of a column, we can't tell if it's supposed to be a number, string, etc.. When this happens, we will show you a warning and you can manually specify the type. #### Number Formats For numeric columns, you can also specify a number format, which controls how the metric is displayed on the front-end. Possible values are `currency`, `time:seconds`, `memory:bytes`, and `memory:kilobytes`. :::note USD by default The default display currency is USD, but you can change it under **Settings** → **General** → **Metric Settings** ::: ## Metric Types There are 7 types of metrics you can create within GrowthBook - Proportion, Retention, Daily Participation, Mean, Ratio, Quantile, and Funnel. ### Proportion Metrics Proportion Metrics measure the percent of users in an experiment who exist in a Fact Table (and match any filters you specify). For example, if you have an Orders Fact Table, you can create a Proportion Metric to see what percent of users purchased something. For this metric, we don't care how many times they purchased, just that they have at least 1 matching row. ### Retention Metrics Retention Metrics are used to measure the retention of users over time with respect to their first experiment exposure. For example, you might want to measure the retention of users who came back to your site at least 7 days after their first experiment exposure. You can use the filters to determine if a user qualifies as retained (using a row filter to only look for certain events or using a user filter to only look for users who completed an action a certain number of times). You then set the retention window, which is the amount of time after the first experiment exposure that a user must do the qualifying actions. You can also use a conversion window to only look for a certain time period after the start of the retention window. For example, if you want to measure the retention of users who came back to your site in the second full week after the experiment exposure, you would set the retention window to 7 days and the conversion window to 7 days. ### Daily Participation Metrics Daily Participation Metrics measure the average percentage of days after experiment exposure that a user is active (i.e., meets the metric criteria). This is analogous to Daily Active Users (DAU) in an experiment context, where you want to measure engagement frequency over time. For each user, GrowthBook: 1. Counts the number of distinct days they match the metric criteria (have rows in the Fact Table that meet your filters) 2. Calculates the number of eligible days since their first experiment exposure (based on the metric window settings) 3. Computes the user's participation rate as: `(matching days) / (eligible days)` 4. Averages these participation rates across all users in the variation The result is displayed as a percentage (0-100%), representing the average daily participation rate for users in that variation. **Example:** If your control variation has two users: - **User 1**: Meets the active criteria on 2 days and has been in the experiment for 5 days → participation rate = 2/5 = 0.4 (40%) - **User 2**: Meets the active criteria on 3 out of 3 days → participation rate = 3/3 = 1.0 (100%) The variation average would be: (0.4 + 1.0) / 2 = 0.7, or **70%**. This means the average daily participation rate is 70% for users in this variation. If you had a conversion or lookback window, the number of eligible days would be limited to the number of days in that window and the participation rate would be calculated based only on that window. **Use Cases:** - Measuring user engagement frequency (e.g., "What percentage of days do users log in?") - Tracking feature adoption over time (e.g., "What percentage of days do users use the new feature?") ### Mean Metrics Mean Metrics are useful when there are more than 2 possible states a user can be in. For example, instead of just purchase/not purchase, you might care about the average number of orders a user makes, the average revenue you earned per user in an experiment, or a count of the distinct types of items a user purchased. For the metric value, there are 4 types of aggregations you can do: - `COUNT(*)` - a count of all rows per user in your Fact Table - `COUNT(DISTINCT DATE(timestamp))` - the distinct number of days user completed the qualifying actions in your fact table (e.g. for a count of active days per user, akin to daily active users [DAU] metrics) - `SUM(column)` - the sum per user of any numeric column in your Fact Table - `MAX(column)` - the max of a numeric column per user in your Fact Table - `COUNT(DISTINCT column)` - the distinct count of a string column per user in your Fact Table. Note: for `COUNT(DISTINCT column)` where you pass an arbitrary column, we use the [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) estimation method to approximate the distinct count while scaling to account for data with many rows and easily allow re-aggregation. However, `COUNT DISTINCT` is still more computationally costly than other aggregation types, so use it only when necessary. This aggregation is not available for SQL Server, MySQL, Postgres, or Vertica. For example, an **Orders per User** metric would use `COUNT(*)` since you want to count the total number of rows. A **Revenue per User** metric would use `SUM` and select the column where the order total is stored. The denominator for Mean Metrics is always the number of users in an experiment variation. So, even though you may be doing `SUM(revenue)` as the aggregation, it will be divided by number of users and you will actually end up with an average value per user, not an overall sum for the variation. ### Ratio Metrics Ratio Metrics let you pick a custom denominator and allow for metrics such as **Average Order Value** (denominator is number of orders) or **Session Duration** (denominator is number of sessions). For both the numerator and denominator, you can select a Fact Table, an aggregation, and optional filters. GrowthBook allows the numerator and denominator to have different Fact Tables, which can open up some really advanced use cases. The types of aggregations supported for ratio metrics are: - `COUNT(*)` - Number of rows in the Fact Table - `COUNT(DISTINCT 'User Identifier')` - Sample size of the experiment variation - `SUM(column)` - Total of a numeric column in the Fact Table - `MAX(column)` - Max of a numeric column in the Fact Table by user - `COUNT(DISTINCT column)` - Distinct count of a string column in the Fact Table by user Under the hood, GrowthBook uses the Delta Method to accurately determine the variance. For example, to create a **Session Duration** metric, you would do the following: - Numerator: `SUM(duration)` from a `Sessions` fact table - Denominator: `COUNT(*)` from a `Sessions` fact table ### Quantile Metrics Quantile Metrics are useful when you care about comparing variations at a quantile (e.g. latency at P99) rather than the mean. These are also called percentile metrics. Learn more about when to use Quantile Metrics [here](/statistics/quantile). One key consideration is `Group by Experiment User before taking quantile?`. In short, if you often think about your metric at the user level, you should aggregate. If you think about your metric at the event level, do not aggregate. For the metric value, the only aggregation available is `SUM(column)`. For `SUM`, you will be able to choose any numeric column in your Fact Table. You have the option to `Ignore Zeros`. For event-level analysis, ignoring zeros entails removing all events from the analysis with value equal to 0. Similarly, for a user-level analysis, removing zeros entails removing all rows corresponding to user-level aggregations that have value 0. When picking `Quantile` you have several default options or can select `Custom`. If you select `Custom` you must input a number between 0 and 1. For example, inputting 0.75 will compare the $75^{\text{th}}$ percentiles across variations. ### Funnel Metrics Funnel Metrics measure the share of exposed units who complete a series of steps in order, such as viewing a product, adding it to a cart, and then purchasing. Instead of a numerator, you define an ordered list of steps, each with its own row filters and conversion window. Each step is measured against all exposed units, not just the units who completed the previous step, which keeps every step directly comparable to your other metrics. The metric's own value is the share of units who reach the last step. See [Funnel Metrics](/app/metrics/funnel) for step settings, conversion windows, optional steps, and the API. ## Filters The real power of Fact Tables comes with filtering. This allows you to easily create metric variants. There are 2 kinds of filtering available - Row Filters - User Filters ### Row Filters Row Filters determine which rows of the Fact Table are included in the metric calculation. You can define multiple Row Filters and they will be ANDed together. Row Filters support a wide variety of operators: - `=` - `!=` - `>` - `=` - ``, `>=`, `between`, `not_between`, and `is not null` for these columns, and a value is either a calendar day (`2024-01-15`) or a minute-precision timestamp (`2024-01-15T09:30`). A calendar day matches the whole day rather than the instant at its midnight, so `= 2024-01-15` includes every row stamped on that date. #### Auto Prompting for Row Filters It's common to have an important "type" column that you use to build out metrics. For example, if your Fact Table is a raw event stream, there will be a column like `event_type` to differentiate between a page view, button click, purchase, etc.. You will almost always want to filter metrics using this column. After creating your Fact Table, you can edit a column and check the “Prompt all metrics” box. This helps GrowthBook better streamline the UI when creating metrics. ![Marking an Event Type column](/images/metrics/inline-filter.png) When you check this box, GrowthBook will run a quick query looking for the top 100 values for this column in the past week. This will be used to add an auto-complete field in the UI when creating a metric: ![Event Type Filter UI](/images/metrics/event-type-filter-metric-form.png) Another example use of this checkbox is for a `Page Path` column in a `Page Views` Fact Table. This will streamline creation of metrics such as "Visited /checkout", "Visited /pricing", and more. ### User Filters User Filters are applied AFTER aggregating by user. Instead of filtering individual rows, you are filtering the total value per user. You can either filter on the `Count of Rows` or the `SUM` of a numeric column. This can be easier to understand with examples: - Count of rows: `>= 3`: Only include users with at least 3 rows in a Fact Table - SUM of a revenue column: `> 100`: Only include users who have spent at least $100 total across all of their orders. Only simple SQL comparison operators are supported: ``, `=`, `=`, ``. User Filters can only be applied to Proportion metrics and the numerator of Ratio metrics (only when using `COUNT(DISTINCT 'User Identifier')`). ## Metric Windows When used in an experiment, we only consider rows of a Metric where the timestamp is greater than or equal to the first time the user was exposed to the experiment. In other words, if someone purchases something before seeing your experiment, it won't be included in the analysis. This behavior is ideal for the vast majority of metrics, but you can change it with the Metric Delay setting if desired (see below). There are three window settings one can use to configure the metric date window. Each of them defines the lower and upper date range of the metric to use for each user: - **None** (default for Fact Metrics) - Lower bound: user's first exposure plus the metric delay - Upper bound: experiment end date - **Conversion Window** (default for Legacy Metrics) - Lower bound: user's first exposure plus the metric delay - Upper bound: the lower bound + the length of the conversion window - **Lookback Window** - Lower bound: the experiment end date minus the lookback window OR the user's first exposure plus the metric delay, whichever is later - Upper bound: experiment end date Here's a graphical representation of these three window types for a hypothetical User 1: ![Metric Windows](/images/metric-windows.png) Here's a second example for a hypothetical User 2, who joins the experiment late. Notice that the conversion window can extend beyond the experiment end date. ![Metric Windows (User 2)](/images/metric-windows-user-2.png) Why might you choose one window over another? **None** - The simplest. Use all data available. This is useful for using as much data associated with users in your experiment and will combine any behavior that is right after experiment exposure as well as long run behavior within the experiment time frame. **Conversion window** - Conversion windows allow you to only look at events that are tied to the first exposure to an experiment. This can help if, for example, you are tracking purchases and you only want to measure the effect of an experiment in a checkout flow on purchases made soon after seeing that checkout flow. Using a conversion window can reduce the noise from user behavior not related to an experiment. However, if you set the window too short, you may not capture users that return a few days later and were influenced by the experiment. **Lookback window** - Lookback windows are good for capturing long run impacts of an experiment on regular behavior like user log ins or page views. They have two main advantages: (1) You can mitigate the novelty effect of an experiment; if you are testing a new recommendation algorithm, at first users may react a certain way to the experiment, but eventually they may adjust and so may their behavior. In these cases, you may just want to look at the last 14 days of an experiment. (2) Lookback windows help you focus on the long run effects of an experiment. Much of experimentation is about building a better product; by focusing on impact of an experiment after it has been live for a week or two, you may get a better picture of the long run impact of launching the experiment. Larger companies who measure long run logged in behavior want to ensure their experiments have lasting effects and often rely on lookback windows to make shipping decisions. However, lookback windows may not be right if you are testing a feature on logged-out or anonymous users and are measuring simple purchase conversions or something similar. In these cases, you might end up with many logged-out users who, in the long run, simply have no metric data associated with them. ## Advanced Settings ### What is the Goal? For the vast majority of metrics, the goal is to increase the value. But for some metrics like "Bounce Rate" and "Page Load Time", lower is actually better. Setting this to "decrease" basically inverts the "Chance to Beat Control" value in experiment results so that "beating" the control means decreasing the value. This will also reverse the red and green coloring on graphs. ### Capped Value Large outliers can have an outsized effect on experiment results. For example, if your normal revenue per user \$40 and someone happens to make a \$5000 order, whatever variation that person is in will be much more likely "win" any experiment because that one order is an outlier. Capping (also known as winsorization) works by ensuring that all aggregate unit (e.g. user) values are no more than some value. So in the above example, if the cap was \$100, the \$5000 purchase will still be counted, but the aggregated value for that user will be capped at \$100 and will have a much smaller effect on the results. It will still give a boost to whatever variation the person is in, but it won't completely dominate all of the other orders and is unlikely to make a winner just on its own. Another way to think about this is that you are slightly biasing your results by truncating large values, but you are reducing variance to prevent the outsized effect of outliers. There are two ways to cap metric values in GrowthBook: **1. Absolute capping** - if set above zero, all aggregated user values will be capped at exactly this value. For example, if the cap is \$100 on total revenue per user, then after we sum all of a users orders up, any user with an aggregate sum of greater than \$100 will be set to \$100. **2. Percentile capping** - when this is set to between 0 and 1, it uses that percentile to select a cap based on the data in your experiment so far. This cap is therefore specific to each experiment and specific to each analysis run in that experiment if new data has come in. It works like so: after we calculate the unit-level aggregate values for all units (e.g. users) during an experiment analysis, we find the specified percentile of these unit-level aggregates and then cap these aggregated values at this percentile. Using the above example, if you were to specify percentile capping with a value of `0.95`, then we find the 95th percentile of total revenue per users (say this turns out to be \$135). We then cap those user-level aggregates at \$135. You can additionally choose to ignore zeros, which will compute the percentile without including any user aggregated zero values. This is useful if you have a lot of zero values and you don't want to have to fine tune the percentile to avoid setting the cap too low. Because the percentile cap depends on the data in your experiment, it can be different from experiment to experiment, or even analysis to analysis. To find out what value was actually used for capping you can do the following: on the Experiment Results tab, click the three dot menu in the top right and select "View Queries". Each percentile capped metric will have a column with the `main_cap_value` that was used to cap that metric and represents the computed percentile of unit-level aggregate values. ### Metric Delay Conversions within the first X hours of being put into an experiment are ignored (default = `0`). This is useful for metrics like "day 2 retention". In that case, if your underlying table reports whether a user is retained on any given day, you could set a metric delay to `24` hours. #### Negative metric delays The metric delay can also be negative to include some conversions **before** a user is put into an experiment. For example, a value of `-2` would mean conversions up to 2 hours before will be included. You might be wondering when this would ever be useful. Imagine the average person stays on your site for 60 seconds and your experiment can trigger at any time. If you just look at the average time spent after the experiment, the numbers will lose a lot of meaning. A value of `20 seconds` might be horrible if it happened to someone after only 5 seconds on your site since they are staying a lot less time than average. But, that same `20 seconds` might be great if it happened to someone after 55 seconds since their visit is a lot longer than usual. Over time, these things will average out and you can eventually see patterns, but you need an enormous amount of data to get to that point. If you set the metric delay to something negative, say `-0.5` (30 minutes), you can reduce the amount of data you need to see patterns. For example, you may see your average go from 60 seconds to 65 seconds. Keep in mind, these two things are answering slightly different questions. `How much longer do people stay after viewing the experiment?` vs `How much longer is an average session that includes the experiment?`. The first question is more direct and often a more strict test of your hypothesis, but it may not be worth the extra running time. ### Bayesian Priors Your organization can set default priors for Bayesian analyses that are used by all metrics. However, you can also set metric specific priors by opening the Edit Metric modal from the Metric page, clicking on Advanced Settings, and turning on the metric override. This will allow you to set a custom prior for that metric. Additionally, you can use experiment metric overrides to further customize these priors for each experiment. You can read more about Bayesian priors on [our statistical details page](/statistics/details). ## Metric Slices Many metrics are easily decomposed, and you may want to learn about the different elements. For example, you may want to see revenue across different product types (e.g. "apparel", "equipment"). Rather than creating and maintaining a separate revenue metric for each product type, metric slices permit you to automatically create distinct revenue metrics for each product type. Metric slices are enabled for Fact Metrics. We offer two key ways to slice your metrics: Auto Slices and Custom Slices. ### Auto Slices These metric slices are defined in your [Fact Table](#fact-tables) and can be selectively enabled per each Fact Metric. When an auto slice is added to a Fact Metric, it is automatically added to all experiments that use the metric. This allows for a standardized deep-dive analyses across your experiments. To set up Auto Slices: 1. Navigate to your Fact Table that has the metrics and the columns by which you want to slice. Ensure the Fact Table SQL is returning the relevant column. If it isn't you can click "Edit SQL" on the Fact Table page, add your column (e.g. `product_type`) to the SELECT statement, and then press "Confirm Changes". Only **string** and **boolean** columns can be used. 2. With your column (`product_type`) in the Fact Table SQL, you now edit the column property in the "Columns" table. Click the edit icon next to your column and then check the "Enable Auto Slices" checkbox in the window that pops up. - For **string** columns, add the slices you want to split your metric by. We run a query to populate the top values for you, and you can update the values found in the data by clicking the "Refresh" button. Slice values here could be "apparel", "equipment", and more. An "other" slice will also be shown if any metric value is attributed to values outside of your selected slices. You can have these levels automatically refresh over time by enabling [auto-update slice levels](#auto-update-slice-levels). - For **boolean** columns, no other slice configuration is necessary—both "true" and "false" slices will be generated. A "null" slice will also be shown for nullable columns if any metric value is attributed to "null". 3. Scroll down to the list of Fact Metrics, or navigate to a Fact Metric from this Fact Table. 4. Click "Edit" and select your slices you want to apply to that metric for all experiments. #### Auto-update slice levels You can enable automatic updates of auto slice levels based on top column values. This keeps your slice levels current with your data without manual intervention. **Fact Table level setting:** In the **Fact Table** modal, check the **Auto-update slice levels** checkbox. When enabled, a scheduled job periodically refreshes auto slice levels from **top values**, scanning the lookback window configured in your organization's **Data Source settings** (default 14 days). The job runs: - Every 7 days by default (configurable via `AUTO_SLICE_UPDATE_FREQUENCY_HOURS` environment variable for self-hosted users) - Automatically when the fact table changes - Manually when you click **Update** in the column modal ![Fact Table modal with auto-update slice levels checkbox](/images/metrics/auto-slice-updates-fact-table.png) #### Locking slice levels When auto-updates are enabled, you can **lock** specific slice levels in the column modal to ensure they are never removed by the auto-update job. This is useful for important segments or product types that you always want to track, even if they're not currently in the top values. In the column modal, each slice level has a **Lock** checkbox. Locked levels are always preserved when the top-values job runs, while unlocked levels may be replaced if they're no longer in the top values. ![Column modal showing slice levels with Lock checkboxes](/images/metrics/auto-slice-levels-locking.png) ### Custom Slices We also allow you to perform custom metric breakdowns across all of your experiment metrics for a single experiment. For example: "For this experiment, I want to see how treatment will affect apparel revenue and average order value in EU". Unlike auto slices, custom slices allow you to create filtering combinations from different columns (e.g., `product_type = 'apparel' and location = 'EU'`). You can add custom slices when setting up your experiment after selecting your metrics, or at any point afterwards. These slices will be run for all metrics in your experiment, but for that experiment only. Custom slices can be created using any non-user-identifier string or boolean column in your Fact Table. (Custom slicing is not limited to auto-slices columns.) ### Viewing metric slices Using our revenue metric as an example, when you run an experiment you will now have the option to split revenue by product type (auto slice) and/or by "apparel and EU" (custom slice). If you click the arrow icon next to "revenue" on the experiment results table, you will see these slices expand under the base metric row (revenue), each slice materialized as its own sub-row. You also can use the Filter button at the top of the results to filter all metrics by slices, or click in to a row for the [Metric Drilldown](/app/experiment-results#metric-drilldown) to view slices with search, sorting, and time-series analysis capabilities. :::note Dimensions versus Slices Unit dimensions and experiment dimensions are attributes specific to an experimental unit (e.g. user). In contrast, metric slices are an attribute specific to a metric. For example, suppose that a user named Romain lives in France, and he buys products from both France and Germany. In a dimension analysis (i.e. a unit dimension analysis), all of Romain's spend will be attributed to France. In a metric slice analysis, only the French products purchased will be attributed to French revenue, and only the German products purchased will be attributed to German revenue. So one user will always be mapped to a single user dimension, but can contribute to multiple metric slices. All users (units) contribute to a metric slice. This provides the most accurate inference, as trying to filter down the denominator can cause bias issues if you allow users to switch between different dimension levels. ::: ## Metric Analysis After creating a metric, you can perform some basic analysis on it. Depending on the metric type, we show you the average value over time, a histogram breakdown, and more. You can customize the date range and population that's included. For more advanced analysis, use our [Product Analytics](/app/product-analytics) feature. ## Metric Groups There are often groups of metrics you want to add together to an experiment. For example, 5 key revenue metrics that you add to anything touching the checkout flow. Define Metric Groups to accomplish this. Add multiple metrics, drag & drop to order them, and add them in bulk to experiments as Goals, Guardrails, or Secondary Metrics. ## Database Cost Optimization For large Fact Tables, it can be important to consider query costs, especially when using a data warehouse like BigQuery which charges based on data scanned. Read our dedicated guide on [Query Optimization](/app/query-optimization) for best practices on reducing query costs across GrowthBook. --- ## Official Resources import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; Official Resources are protected resources that **can only be modified by users with appropriate permissions**. These resources have a checkmark badge in the app to indicate their protected status. This feature is useful for resources that are externally managed or need to remain consistent across teams. ![Example of an official metric](/images/metrics/official-metric.png) ## Use Cases Official Resources are particularly useful for: - **Governance**: Organizations want to prevent unauthorized changes to critical resources - **External System Management**: Resources are managed by external systems (like a version control system) - **Cross-Project Consistency**: Resources need to be identical across multiple projects - **Compliance**: Resources must be managed through specific, auditable processes ## Official Resource Types - Fact Tables - Fact Metrics - Fact Filters - Segments - Dimensions ## 3 Ways to Create Official Resources ### 1. GrowthBook UI Create Official Resources in-app by marking them as "Official" during creation. Only users with the `ManageOfficialResources` policy can apply this designation. ### 2. REST API When creating resources via the REST API, use the optional `managedBy` property: - **`managedBy: "api"`**: The resource is not editable in the app but can be modified via the REST API. This is ideal for resources managed by external repositories or systems. - **`managedBy: "admin"`**: The resource can be modified in the app or via the REST API, but only by users with the `ManageOfficialResources` policy. ###3. config.yml When resources are created via the `config.yml` file, they are automatically marked as "Official" and are not editable in the app. To make changes to these resources, you must update the `config.yml` file. :::note Restrictions on creating Official Resources Currently only Segments, and Dimensions can be created via the `config.yml` file, Fact Tables, Fact Metrics, and Fact Filters can only be created via the UI or REST API. ::: [Learn more about updating metrics with `config.yml`](/self-host/config) ## Managing Official Resources ### Marking Existing Resources as Official Users with the `ManageOfficialResources` policy can mark existing resources as Official in the app, which prevents other users from modifying these resources. ### Unmarking Resources as Official Removing the "Official" status from a resource is possible for users with the `ManageOfficialResources` policy, but the process depends on how the resource was created. - If a resource was created via the `config.yml` file, it cannot be unmarked as "Official." Instead, you must remove the resource from the `config.yml` file, and then create a new version of the resource in the app or via the API. - If a resource was created via the REST API and has `managedBy: "api"`, you will need to update the resource via the REST API and remove the `managedBy` property. - If a resource was created in the app or via the REST API (and has `managedBy: "admin"`), it can be unmarked as "Official" in the app or via the REST API. :::note Limited support for marking Official Resources At this time, only `Fact Metrics`, and `Fact Tables` can be marked or unmarked as "Official" in the app. We are actively working on adding this feature to other resource types. ::: ### Required Permissions To manage Official Resources, users need the `ManageOfficialResources` policy. By default, this policy is only available to users with the `admin` role. However, Enterprise organizations can create custom roles and assign the `ManageOfficialResources` policy to them. For more information about roles and permissions, see [User Permissions](/account/user-permissions). --- ## An overview of the GrowthBook Platform import ButtonCard from '@site/src/components/ButtonCard' # What is GrowthBook? GrowthBook is a modular platform. You can use it for either Feature Flags, Experiment Analysis, or both. [**Feature Flags**](/app/features) you create in the GrowthBook UI are published to our API as a JSON file. Pass this JSON into our SDKs and use the feature flags throughout your code. If a feature is running as part of an experiment, we'll fire a tracking callback in the SDK so you can record that event in your data warehouse or analytics platform for later analysis. [**Experiment Analysis**](/app/experiment-configuration) queries your data warehouse for raw experiment data and runs it through our stats engine to produce a report. No raw user-level events or PII are ever sent to GrowthBook. We only get back aggregate info instead (sums, sums of squares, etc.). ![GrowthBook: How it Works Diagram](/images/feature-flagging-experimentation-diagram-766.png) ## Feature Flags [Feature flags](/app/features) are a very powerful developer tool. They give you deep control over how and when new functionality is released to your users. GrowthBook supports 4 types of features: - **Boolean (on/off)** - **Number** - **String** - **JSON** (with JSON Schema validation) Using features in your code is easy. Here's an example from our Javascript SDK: ```js // For boolean features if (growthbook.isOn("my-feature")) { // ... Do something } // For number, string, and JSON features const value = growthbook.getFeatureValue("other-feature", "fallback"); ``` GrowthBook also supports multiple environments, so you can, for example, have a feature enabled in **dev** but not **production**. You can also change the value of a feature with **Override Rules**. The following types of rules are supported: - **Forced Value** - Choose a subset of users based on targeting attributes and assign them all the same value - **Percentage Rollout** - Use random sampling to roll out a new feature value to a percent of users - **Safe Rollout** - Gradually ramp up traffic while monitoring guardrail metrics and automatically roll back if things go wrong - **Experiment** - Run a controlled hypothesis test between 2 or more feature values - **Multi-Arm Bandit** - Dynamically allocate traffic to different feature values based on performance All features for an environment are packaged together into a single JSON file. All you need to do is pass this into our [SDKs](/lib) along with [user targeting attributes](/features/targeting). Read more about [features here](/app/features). ## Experiment Analysis GrowthBook needs to connect to your [SQL Data Warehouse](/warehouses) in order to query experiment results. GrowthBook is extremely flexible and can support almost any schema structure with a little bit of configuration. ![Data warehouses supported by GrowthBook](/images/GrowthBook-supported-DB-766.png) Once connected to a data source, you need to define [Fact Tables and Metrics](/app/metrics). Fact Tables are SQL queries that select raw data from your warehouse. Metrics are built on top of these Fact Tables and define what you want to measure in your experiments. For example, you might have an **Orders** fact table that selects one row per order, and then metrics for **Average Order Value**, **Total Revenue**, and **Number of Orders**. GrowthBook supports many types of metrics: - **Proportion** - A simple conversion rate (e.g. `signed up`, `purchased`). - **Mean** - The average value across experiment units (e.g. `revenue`, `page views`). Supports several different aggregation types: sum, count, max, and count distinct. - **Ratio** - A ratio of two values (e.g. `pages per session`, `revenue per order`). - **Quantile** - The value at a specific percentile (e.g. `P99 Latency`). - **Retention** - The percentage of experiment units who return after a delay (e.g. `day 7 retention`). - **Funnel** - The percentage of experiment units who complete a series of steps, with a per-step breakdown of where units drop off (e.g. `signup funnel conversion`). Once you create a library of metrics, you can reuse them across multiple experiments. GrowthBook has an extremely powerful and highly configurable statistics engine for analyzing experiment results. Below are some of the key features and options: - **Frequentist or Bayesian** - Choose the statistical framework that best fits your needs - **Sequential Testing** - Always-valid p-values that allow you to safely peek at results as often as you like - **CUPED** - Use pre-experiment data to reduce variance and improve sensitivity - **Sample Ratio Mismatch (SRM)** - Automatically detect when experiment traffic is not being properly split between variations - **Multiple Comparison Corrections** - Control for either false positive rate (Holm-Bonferroni) or false discovery rate (Benjamini-Hochberg) - **Different Baselines** - Choose any variation as the baseline for comparison, not just control - **Custom Priors** - For Bayesian stats, set custom prior distributions for each metric - **Power Calculator** - Estimate the required runtime based on historical data before launching your experiment - **Difference Types** - View metric differences as either relative lift, absolute lift, or scaled impact ![Results Table](/images/results-table.png) Your data team can drill down into results by custom [dimensions](/app/dimensions), view the raw SQL that GrowthBook is running on your data warehouse, and export results to a Jupyter notebook for even deeper analysis. Read more about [experiments here](/experiments). ## Use Cases There are typically three reasons that teams use GrowthBook. ### 1. Full Experimentation Platform In this use case, companies use Feature Flags and our SDKs to run experiments in their applications. Then they use our Experiment Analysis to look at the results and decide on a winner. This is best for companies that want an all-in-one solution and seamless developer experience. ### 2. Feature Flags Only In this use case, companies don't run experiments at all and just use GrowthBook feature flags within their engineering team. This is best for companies that don't have enough traffic to run full experiments, but still want all of the benefits that feature flags provide. It's also good for companies that know they will want to run experiments in the future and want to start instrumenting their applications today to get ready. ### 3. Experiment Analysis Only In this use case, companies are already running experiments and analyzing results usually with either a home-built reporting system or by manually creating Jupyter notebooks. They use GrowthBook to automate and improve the analysis process to save time and make better decisions. This is best for companies that already have a robust process for running experiments and just need a little help analyzing results at scale. ## Next Steps } color="default" /> } /> } color="default" /> --- ## Dashboards import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from "@site/src/components/CommercialFeature"; # Dashboards Product Analytics Dashboards are general-purpose dashboards built from the same explorations you create in the [Explorer](/app/product-analytics/explorer). Use them to monitor KPIs, document findings for stakeholders, or give a team a single place to track the metrics they care about. :::note Product Analytics Dashboards are separate from [Experiment Dashboards](/app/experiment-dashboards), which are scoped to a single experiment. The block types and configuration overlap, but the two surfaces are managed independently. ::: ## Creating a Dashboard 1. Navigate to **Product Analytics → Dashboards**. 2. Click **Create Dashboard** in the top right. 3. Configure the dashboard's settings (see below) and click **Create**. Once created, you'll be dropped into the dashboard editor where you can start adding blocks. ### Dashboard Settings When creating or editing a dashboard, you can configure: - **Name** — A descriptive title for the dashboard. - **Projects** — Limit the dashboard to one or more [projects](/account/user-permissions#project-specific-permissions). Leave empty to make it available across all projects. - **Auto-update** — When enabled, GrowthBook refreshes the dashboard's data on a schedule (see [Update Schedules](#update-schedules)). - **View access** — Who can open the dashboard: - **Organization members** — Anyone in your organization can view it. - **Only me** / **Owner only** — Only the owner (and admins) can view it. - **Edit access** — Who can modify the dashboard: - **Any organization members with editing permission** — Other users with dashboard edit permission can change the contents. - **Only me** / **Owner only** — Only the owner (and admins) can edit it. When **View access** is set to private, **Edit access** is forced to private as well — you can't have a dashboard nobody can see but other people can edit. :::note Sharing dashboards across the organization requires the **Share Product Analytics Dashboards** commercial feature, which is included on Enterprise plans. On Pro plans, dashboards are limited to private (owner-only) view and edit access. ::: ### Update Schedules When **Auto-update** is enabled, you can choose how often the dashboard refreshes: - **Stale-based** — Refresh whenever the data is older than _N_ hours. Defaults to 6 hours; can be set between 1 and 168 (one week). - **Cron** — Refresh on a custom cron schedule. The minimum frequency is once per hour. The modal validates the cron string in real time and shows a human-readable description (e.g., `0 0 */2 * *` → "at 12:00 AM, every 2 days"). You can also refresh a dashboard manually at any time using the **Update** button on the dashboard view (see [Refreshing a Dashboard](#refreshing-a-dashboard)). ## The Dashboard Editor Dashboards are built from a stack of blocks. Click **Edit** on a dashboard to open the editor. The editor has three areas: - **The block list** on the left lets you reorder blocks via drag-and-drop. Hovering over a block reveals quick actions for editing, duplicating, and deleting. - **The dashboard canvas** in the middle shows the live preview of the dashboard with all blocks rendered. - **The configuration sidebar** on the right shows settings for the currently selected block (data source, chart type, dimensions, etc.). Edits go into a **draft state** — none of your changes are persisted until you click **Save**. Use **Cancel** to discard the draft and return to the published version. The editor also keeps the rest of your in-flight work in memory, so you can switch blocks without losing changes. ### Adding Blocks Click **Add Block** at the bottom of the block list (or the `+` button between two existing blocks) and pick a type from the dropdown. The dropdown is grouped to make blocks easy to find. ## Dashboard Blocks The following block types are available on Product Analytics Dashboards. Each block has its own configuration sidebar with controls tailored to that block type. | Block | Purpose | | :----------------------- | :-------------------------------------------------------------------------------------------------------------------- | | **Markdown** | Add formatted text, links, and images for context, headings, summaries, or section dividers. | | **Metric Explorer** | Visualize one or more existing metrics. Same configuration as the [Metric Explorer](/app/product-analytics/explorer). | | **Fact Table Explorer** | Aggregate a fact table by counts, distinct units, or a sum. Configure dimensions and chart type. | | **Data Source Explorer** | Query a timestamped warehouse table without first defining a fact table. | | **Custom SQL Query** | Render saved queries from [Custom SQL Reports](/app/product-analytics/explorer#custom-sql-reports). | ### Markdown blocks Markdown blocks render standard Markdown — headings, lists, links, images, code blocks, and tables. Use them to: - Title sections of the dashboard. - Provide context, methodology, or caveats next to a chart. - Embed images (such as wireframes or product screenshots). - Link out to related Slack threads, Notion pages, or other documentation. ### Explorer blocks The Metric, Fact Table, and Data Source Explorer blocks share the same configuration UI as the standalone [Explorer](/app/product-analytics/explorer). The block sidebar shows: - A condensed toolbar with **Chart Type**, **Date Range**, and (for time-series charts) **Date Granularity**. - The full sidebar — data source, values, row filters, group-by dimensions, and Show As. Each block runs its own query against the warehouse, so a single dashboard can mix data from multiple data sources or fact tables. ### Custom SQL Query blocks A Custom SQL Query block is bound to a saved query from [Custom SQL Reports](/app/product-analytics/explorer#custom-sql-reports). The block can show: - One or more visualizations attached to the saved query (Bar, Line, Area, Scatter, Big Value, or Pivot Table). - The raw results table — useful when the value of the report is the data itself rather than a chart. The order of items in the block matches the order configured on the saved query, and you can reuse the same saved query across multiple dashboards. ## Saving an Exploration to a Dashboard You can build a chart in the Explorer and add it to a dashboard in one step: 1. Configure the exploration in the Explorer. 2. Click **Save to Dashboard** in the top-right of the configuration sidebar. 3. Pick an existing dashboard or create a new one. This saves the exploration as a **Metric Explorer**, **Fact Table Explorer**, or **Data Source Explorer** block on the destination dashboard. See [Save to Dashboard](/app/product-analytics/explorer#save-to-dashboard) for the full details on the modal. ## Refreshing a Dashboard Each dashboard has an **Update** button in the top right. When you click **Update**, GrowthBook re-runs every block's query against the warehouse. While the refresh is in progress, you'll see: - A **progress bar** under the button showing how many of the queries have finished. - A live status text — `Updated 5m ago`, `One or more queries failed`, or `Update Failed` — with a tooltip describing the error if one occurred. If auto-update is enabled, a **lightning icon** appears next to the status, and the tooltip shows when the next automatic refresh will run. :::note Refreshing the dashboard requires the **Create Analyses** permission and read access to every data source the dashboard touches. ::: ## Sharing and Collaboration :::note Only **Enterprise** organizations can publish dashboards for organization-wide viewing. On **Pro** plans, dashboards are limited to private (owner-only) view and edit access. ::: Once a dashboard is published to the organization, anyone with view access can: - Open the dashboard. - See the latest auto-refreshed data (or trigger a manual refresh). - Click **Share...** to copy a direct link. To control sharing on an existing dashboard, use the **More** menu (⋯) on the dashboards list and choose **Share...**. Only the owner and organization admins can change view and edit access. ### Reassigning Ownership Editing a dashboard you own (or that you administer) lets you change the **Owner**. Reassigning ownership is useful when a teammate leaves the company or hands a dashboard off to a different team. ### Duplicating Dashboards Use the **Duplicate** option in the **More** menu to create a copy of an existing dashboard. This is useful when you want to use one dashboard as a template for another (for example, copying a "Marketing KPIs" dashboard to start a "Growth KPIs" version). Duplicates inherit all blocks but are owned by the user who duplicated the dashboard. ## Permissions Dashboard actions are controlled by per-project permissions: | Action | Required permission | | :------------------------------ | :------------------------------------------------------------- | | View a dashboard | View access on the dashboard + access to its projects | | Create a dashboard | Create General Dashboards | | Edit a dashboard | Update General Dashboards (and edit access on the dashboard) | | Refresh dashboard data | Create Analyses + Run SQL Explorer Queries on each data source | | Delete a dashboard | Delete General Dashboards (only the owner or org admins) | | Change view/edit access | Owner or organization admin | | Share dashboards across the org | Enterprise plan (Share Product Analytics Dashboards feature) | Private dashboards (where edit access is set to **Owner only**) can only be edited and deleted by the owner and organization admins, regardless of other permissions. See [User Permissions](/account/user-permissions) for the full set of dashboard-related permissions. ## Best Practices - **Group by project** — Scope dashboards to a project so the right people see them in the dashboards list and so they show up under the project filter. - **Lead with markdown** — Use a Markdown block at the top of the dashboard to explain the dashboard's purpose, the data sources it uses, and any caveats. Stakeholders shouldn't have to guess what they're looking at. - **Mix explorations and SQL** — Cover the common cases with Metric and Fact Table explorers, and drop into Custom SQL Reports for the long-tail questions that don't fit a structured explorer. - **Keep auto-update enabled** — Pick a refresh cadence that matches your team's data freshness needs (6 hours for daily standups, nightly cron for weekly reviews) so stakeholders always see fresh data when they open the dashboard. - **Reuse saved queries** — A single saved SQL query can be referenced by many dashboards. Update the query once and every dashboard that links to it gets the new logic. --- ## Explorer import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Explorer The Explorer is an ad-hoc analysis surface for your warehouse data. Use it to chart a metric over time, slice a fact table by a property, query a raw table, or ask a question in natural language. :::note The Explorer is currently in **Beta**. The set of supported chart types, dimensions, and data sources is still expanding. ::: ## Choosing an Explorer Open **Product Analytics** in the left sidebar. From the landing page, you can: - **Ask a question with AI Chat** — Type a question like "How many signups did we have last week, broken out by country?" and let GrowthBook generate the exploration for you. - **Choose an explorer manually** — Pick the type that matches the shape of your data: | Explorer | When to use it | | :-------------- | :------------------------------------------------------------------------------------------------------------- | | **Metrics** | Visualize one or more existing GrowthBook metrics over time. | | **Fact Table** | Aggregate a fact table — counts, distinct units, or a sum of a numeric column. | | **Data Source** | Query any timestamped table in your warehouse without first defining a fact table. | | **Funnel** | Measure how many users complete an ordered sequence of steps and where they drop off. See [Funnels](#funnels). | | **Custom SQL** | Write your own `SELECT` query and visualize the result. Opens [Custom SQL Reports](#custom-sql-reports). | Each explorer opens with the same layout: a chart on the left, a configuration sidebar on the right, and a toolbar above the chart for chart type, date range, and granularity. ![Explorer with chart and configuration sidebar](/images/product-analytics/explorer-overview.png) ## Configuring an Exploration The sidebar controls everything that goes into the chart. Whenever you change a setting, the chart goes "stale" — an amber dot appears next to the **Update** button. Click **Update** to re-run the query with the new configuration. ### Data Source Pick the data source you want to query from the dropdown at the top of the chart area. The dropdown is filtered to data sources you have permission to query in the current project. If you change the data source mid-exploration and you've already configured values, GrowthBook prompts for confirmation since switching clears your current dataset. The Fact Table and Data Source explorers only show tables that belong to the selected data source. ### Values Values are the things being measured. The available value types depend on which explorer you're in. #### Metric Explorer Add one or more existing fact metrics. Metrics are grouped in the dropdown: - **Official Metrics** — metrics managed by GrowthBook or imported from a [managed source](/app/metrics). - **Other** — metrics created and edited directly in your organization. You can chart multiple metrics on the same exploration as long as they're _compatible_: - **Ratio metrics** can't be combined with non-ratio metrics in the same chart. - **Quantile metrics** can't be combined with other metric types. - All other metric types can be mixed freely. Incompatible metrics show up grayed out in the dropdown with a tooltip explaining the restriction. #### Fact Table Explorer Pick a fact table from the dropdown, then add one or more values. Each value has: - **Value type** — `Unit Count` (distinct users), `Count` (rows), or `Sum` of a numeric column. - **Value column** (when value type is `Sum`) — choose which numeric column to sum. Only `number`-typed columns from the fact table are listed. #### Data Source Explorer Pick a table from your warehouse, then choose its **Timestamp Column** — GrowthBook will try to infer this from the column types, but you can override it. After that, the value configuration is the same as the Fact Table explorer (`Unit Count`, `Count`, or `Sum`). If you haven't built an information schema for the data source yet, the explorer prompts you to generate one. Schema building runs in the background; if it takes more than a few minutes, you're free to leave the page and come back later. ### Row Filters Each value in the Fact Table and Data Source explorers can have its own row filters to narrow which events contribute to it. ![Row filters on a value card](/images/product-analytics/explorer-filter.png) A filter has three parts: 1. **Column** — a column on the fact table or data source. JSON sub-fields appear as `column.field`. You can also pick `SQL Expression` to write a raw SQL `WHERE` clause, or `Saved Filter` (Fact Tables only) to reuse a filter defined on the fact table. 2. **Operator** — depends on the column's data type. Common operators include `=`, `!=`, `in`, `not_in`, `is_null`, `not_null`, `>`, `=`, ` ![AI Chat with conversation history and an embedded chart](/images/product-analytics/explorer-ai-chat.png) ### What it can do The AI agent has access to a small set of tools and will narrate what it's doing as it works: - **Search** the data sources, fact tables, and metrics available to you. - **Inspect data shape** — list available columns and column types on a table. - **Inspect values** — sample top values for a column. - **Run an exploration** — produce a chart inline in the conversation. You can click into any chart to open it in the Explorer for further editing. - **Get a snapshot** — pull recent values when reasoning about your data. ### Conversations and history Conversations are persisted, so you can come back later and pick up where you left off. The sidebar lists previous conversations and lets you start a new chat at any time. While the agent is generating a response, you can cancel mid-stream. You can also choose which model to use from the dropdown — GrowthBook defaults to the model configured in **Settings → General → AI**. ### Requirements AI Chat requires both: - **AI features enabled** for your organization (configured under **Settings → General → AI**). - The **AI Suggestions** commercial feature on your plan. If either is missing, the AI Chat input is disabled with an inline explanation. ## Custom SQL Reports For analysis that doesn't fit the structured explorers, use **Custom SQL Reports** at `/sql-explorer`. The page lists all saved queries in your organization, scoped to the current project. ### Writing a query Clicking **New SQL Report** opens the SQL Explorer modal. You get: - **A schema browser** showing databases, schemas, tables, and columns for the selected data source. - **A SQL editor** with auto-complete on table and column names, and a "Format SQL" button (when supported by the dialect). - **A results panel** that shows the first 1,000 rows along with the rendered SQL and query duration. Results can be downloaded as CSV. - **Optional AI assistance** for generating or modifying SQL (requires the AI Suggestions commercial feature). :::note SQL Explorer enforces read-only `SELECT` queries. The query is rejected client-side if it doesn't start with `SELECT` or `WITH` (after stripping comments and string literals). Results are capped at **1,000 rows**. ::: ### Visualizations Once a query has run successfully, switch to a visualization tab to render the results as a chart. Saved queries can have any number of visualizations attached. Available chart types are: - **Bar** - **Line** (with an option to anchor the y-axis to zero) - **Area** - **Scatter** - **Big Value** — a single number with formatting options (`shortNumber`, `longNumber`, `currency`, `percentage`, `accounting`). - **Pivot Table** Each visualization configures its own X axis (or X axes for pivot tables), Y axis, optional dimensions, and chart-specific filters. ### Saving and reusing Saving a query requires the **Save SQL Explorer Queries** commercial feature, available on Pro and Enterprise plans. Saved queries can be: - Re-opened from the Custom SQL Reports list. - **Linked to dashboards** via the **Custom SQL Query** block type. A single saved query can power multiple dashboards. ## Permissions The actions available in the Explorer depend on your role and project assignments: | Action | Required permission | | :------------------------------ | :---------------------------------------------------------- | | Use the Metric explorer | Run Metric Queries | | Use the Fact Table explorer | Run Fact Queries | | Use the Data Source explorer | Run Fact Queries + Run Schema Queries | | Use Custom SQL | Run Fact Queries | | Save SQL Explorer queries | Create SQL Explorer Queries (Pro or Enterprise plan) | | Save explorations to dashboards | Create / Update General Dashboards (Pro or Enterprise plan) | See [User Permissions](/account/user-permissions) for a full breakdown of what each role can do. --- ## Product Analytics # Product Analytics Product Analytics turns the data and metrics you've already configured in GrowthBook into a general-purpose analytics workspace. Visualize KPIs, explore raw events, run ad-hoc SQL, and share curated views with your team — without launching an experiment. Product Analytics is made up of two surfaces: - **[Explorer](/app/product-analytics/explorer)** — Build one-off charts from your metrics, fact tables, or raw warehouse tables. Ask questions in natural language, or configure explorations manually. - **[Dashboards](/app/product-analytics/dashboards)** — Combine multiple explorations, SQL queries, and markdown into a single shareable view that auto-refreshes on a schedule. Both surfaces read from the same [data sources](/warehouses), [fact tables, and metrics](/app/metrics) you already use for experimentation, so there's no separate event pipeline or instrumentation to set up. ## Plan Availability | Capability | Plan | | :-------------------------------------------------- | :---------------------------------------------------- | | Explorer (Metrics, Fact Table, Data Source) | All plans | | AI Chat in the Explorer | Pro+ (requires the AI Suggestions commercial feature) | | Custom SQL Reports (saved queries + visualizations) | Pro+ | | Product Analytics Dashboards (private) | Pro+ | | Sharing dashboards across the organization | Enterprise | The Explorer is currently in **Beta**. We're actively expanding the supported chart types, dimensions, and data sources. ## Getting Started 1. Connect a [data source](/warehouses) so GrowthBook can run queries against your warehouse. If you don't have a warehouse yet, you can use a [Managed Warehouse](/app/managed-warehouse) to get started without provisioning your own. 2. Define [fact tables and metrics](/app/metrics) — these power the Metric and Fact Table explorers, and they're what dashboard blocks visualize. 3. Open **Product Analytics** in the left sidebar to start exploring. The first time you point an explorer at a data source, GrowthBook builds an _information schema_ — an index of databases, schemas, tables, and columns — so the Fact Table and Data Source explorers can show you what's available. This is a one-time, per-data-source step that runs in the background. ## What's Next - **[Build your first chart](/app/product-analytics/explorer)** in the Explorer. - **[Assemble a dashboard](/app/product-analytics/dashboards)** that combines multiple charts, SQL queries, and context. - **[Connect Custom SQL Reports](/app/product-analytics/explorer#custom-sql-reports)** for analyses that don't fit a structured explorer. --- ## Quick Start Guide to GrowthBook import ButtonCard from '@site/src/components/ButtonCard' import MaxWidthImage from '@site/src/components/MaxWidthImage' import { StepperStep, Stepper } from '@site/src/components/Stepper' # Quick Start Guide This guide walks you through the basics of a full integration of GrowthBook for both feature flagging and experimentation. ## Set up GrowthBook You can either **self-host** GrowthBook or use our **Cloud**. For the easiest setup, [sign up for a free Cloud account](https://app.growthbook.io). If you prefer to self-host, follow the [instructions here](/self-host) or on our GitHub page. After creating your account, it will walk you through the initial setup steps, including integrating one of our SDKs into your application. :::tip Press `Cmd+K` (Mac) or `Ctrl+K` (Windows and Linux) anywhere in GrowthBook to open the [command palette](/command-palette) and jump to any feature, experiment, metric, page, doc, or REST API endpoint. ::: ## Feature Flags ### Integrate the GrowthBook SDK As part of the initial setup, you should have created an initial SDK Connection. If not, you can do so now by going to **SDK Configuration** → **SDK Connections**. Then, click **Add SDK Connection**. Follow the directions based on your application's language and platform. The [full SDK docs](/lib) are also available if you want to dive deeper. ### Create a feature flag On the **Features** page, create your first feature flag. ![Create Feature](/images/features/feature-create-feature-1.png) The **Feature Key** is what you will reference in your application and cannot be changed later. Feature flags in GrowthBook are robust, supporting advanced [targeting](/features/targeting), powerful [rule evaluation](/features/rules), [prerequisite features](/features/prerequisites), JSON schema validation, and more. In this quick start, we'll just stick with a simple boolean feature flag that is always `on` or `off` for everyone. ### Use the feature in your application When you create your first feature, you'll see instructions on how to use it in your application. Here's an example using our JavaScript SDK: ```js if (gb.isOn("my-feature")) { console.log("It's On!") } ``` It really is that simple to get started! For next steps, we recommend reading our [Feature Flag Basics](/features/basics) page, which goes into more depth. Feature flags are the foundation for powerful experimentation. In the next section, we show you how to use them to run no-code experimentation using our Visual Editor. Need more advanced options? Dive into our [complete guide for code-based experimentation](/features/feature-flag-experiments). ## Experiment Analysis ### Set up your data source GrowthBook connects to a data warehouse to query your experiment data. You have two options: **Fastest path: [Managed Warehouse](/app/managed-warehouse).** If you don't have an existing data warehouse, our Managed Warehouse gets you running quickly. We provision a fully managed database, create starter metrics, and provide built-in event tracking. No infrastructure setup required. **[Bring Your Own Warehouse](/warehouses).** If you already have event data in BigQuery, Snowflake, Databricks, Redshift, or another warehouse, you can connect it directly. GrowthBook supports almost any schema structure with a bit of configuration and only needs read-only access. :::tip Not sure which to pick? See our [data path comparison](/app/choose-data-path) for help deciding. ::: To get started, go to **Metrics and Data** → **Data Sources** and pick your path. ### Add a metric or two GrowthBook needs to know what metrics you want to measure for your experiments. This starts by creating one or more Fact Tables using SQL, which selects raw data from your warehouse. Metrics are then built on top of these Fact Tables. Go to **Metrics and Data** → **Fact Tables**. Click **Add a Fact Table** and edit the default SQL if needed. Here's an example SQL query for an Orders fact table. ```sql SELECT user_id, timestamp, qty, amount FROM orders ``` Once the fact table is created, you can easily add metrics on top of it. For example, consider a simple proportion metric, which measures the percentage of experiment users who complete an action at least once. In this case, it'd measure the percentage of users who made a purchase. Metrics in GrowthBook are powerful and have many advanced settings. Read about them on our [Metrics page](/app/metrics). ### View experiment results Go to the **Experiments** page. If you implemented an experiment using feature flags, your experiment will already be listed. Click through to it. If you ran your experiment outside of GrowthBook instead, that's fine, too! Add a new experiment and select the **Import Existing** option. Once inside your experiment, go to the **Results** tab. Edit your analysis settings and pick the metrics you created earlier. The results will show up in a table like this: ![Results Table](/images/results-table.png) We have an [entire page in our docs](/app/experiment-results) just about these results and how to interpret them. ## Next steps } /> } /> } /> --- ## Experiment Templates import CommercialFeature from '@site/src/components/CommercialFeature'; # Experiment Templates Experiment Templates provide standardized configurations for experiment creation across your team. By reducing setup overhead and eliminating common boilerplate, templates enable faster experiment initialization and consistency in experimental design. ## Creating Experiment Templates Experiment Templates allow you to pre-configure: - Experiment metadata and details - Traffic allocation - Targeting rules (saved groups, attributes, and prerequisites) - Data source with goal, secondary, and guardrail metrics ![Creating an Experiment Template](/images/experiment-template-create.webp) ### Method 1: Start From Scratch 1. Go to Experiments 2. Click Add 3. Select Create Template 4. Configure your template settings ### Method 2: Start From an Existing Experiment 1. Open the existing experiment 2. Click the overflow menu (⋮) in the upper right 3. Select Save as template… 4. Following the configuration prompts Use your Experiment Template by selecting it during experiment setup. Your configuration will be automatically applied ✨ ![Using an Experiment Template](/images/experiment-template-new-experiment.webp) ## Managing Templates - To view all templates, go to **Experiments → Templates.** - Use the overflow menu (⋮) to edit, duplicate, or delete a template. - Click any template name to view a list of all experiments currently using that template. - Changes to a template only apply to new experiments—existing experiments remain unchanged. ![Managing Experiment Templates](/images/experiment-template-manage.webp) ## Require Template Usage To require template usage for all new experiments: 1. Navigate to **Settings → General → Experiment Settings** 2. Enable **Experiment Templates** ![Require Template Usage](/images/experiment-template-enforce.webp) :::note Support for Bandits Templates coming soon. ::: --- ## Make Changes to Experiments # Make Changes to Experiments We allow you make changes to an experiment's targeting while it is running. This is especially useful if you want to start running on a small percent of traffic and then gradually ramp up exposure. To make a change to a running experiment, click the "Make Changes" button on the top of the experiment page. You will then be asked what sort of change you want to make, with options ranging from targeting changes to traffic and variation weight changes. You may also make multiple changes at once by choosing "advanced". Based on what you are changing, we will provide a list of possible release plans for that change and provide a recommended plan to use. The UI should guide you to make the correct recommended choice in most cases, but you are able to change this if desired. When changing a **single targeting or traffic setting**, your release plan options may include: - New phase, re-randomize traffic - Same phase, apply changes to everyone - Same phase, apply changes to new traffic only (Sticky Bucketing enabled only) Note that potentially unsafe options per your specific changes may be removed (often "Same phase" options are unsafe for certain types of changes). When using **"advanced"** to make multiple targeting or traffic changes at once, all above options will be made available (including potentially unsafe ones), as well as an additional option: - New Phase, re-randomize traffic, block bucketed users (Sticky Bucketing enabled only) If you are not making targeting or traffic changes and simply want to **start a new phase**, the release plan options will be: - New phase, re-randomize traffic - New phase, do not re-randomize ## Update the existing phase This modifies the experiment in-place without resetting results or re-randomizing users. This provides the best user experience for your users and you can re-use data that was already collected, which can reduce the time the experiment needs to run. However, if you're not careful, this can introduce significant bias and data quality issues into your results. A few categories of changes are broadly considered "safe" and if you are only making these changes, this is the approach we recommend going with. The "safe" changes are: - Increasing the percent of people included in the experiment - Removing a condition from an experiment (e.g. going from "US visitors only" to "All visitors") - Removing an experiment from a namespace All of these changes have something in common - they are simply increasing the number of users exposed to the experiment. There is no effect on existing users in the experiment. If you have Sticky Bucketing enabled, you may also elect to apply changes to new traffic only, leaving already-bucketed users in their existing (sticky) buckets. An example of this scenario could be decreasing the percent of people included in the experiment for all new incoming users, but leaving existing users in their existing buckets. ## Start a new phase This creates a brand new phase of the experiment. All data collected until this point is excluded from the analysis and you start fresh (nothing is deleted from your data warehouse, we just hide old data from the results). In most cases, you will also want to re-randomize traffic. This will cause everyone - including existing experiment users - to get assigned a new random variation. This can be a disruptive user experience since many people will switch from Control to Treatment (or vice versa). Why would you ever want to do this? Some changes you make completely invalidate past results so this lets you cleanly separate the data analysis from before and after you make the change. Re-randomizing traffic can also eliminate [carryover bias](/kb/experiments/carryover-bias) and make your results more reliable and accurate. We recommend this approach for any change that is not considered "safe" (listed above). This can include (but not limited to): - Changing the traffic split (weights) between variations - Adding a new targeting condition - Decreasing the percent of people included Because of how disruptive this can be, it's best to plan ahead before starting an experiment. It's better to start with more conservative targeting and scale up than the reverse (starting big and scaling back). As before, if you have Sticky Bucketing enabled, you have some additional options about how to treat already-bucketed users. By default when starting a new phase, these bucketed users will be reassigned (their sticky bucket will be cleared). You may instead choose to block these users from the experiment going forward (in which case they will still see the control). This strategy is only available in the "advanced" mode. --- ## Customizable Pre-Launch Checklists import CommercialFeature from '@site/src/components/CommercialFeature'; # Customizable Pre-Launch Checklist Use GrowthBook's customizable pre-launch checklists to ensure all requirements are met before running an experiment. This guide explains how to create and manage these checklists. ![The prelaunch checklist menu with text explaining how to add a new task](/images/prelaunch-checklist.png) :::tip Why pre-launch checklists are important Learn more about [why pre-launch checklists are important](https://blog.growthbook.io/custom-prelaunch-checklists/). ::: ## How to Create a Pre-Launch Checklist Go to **Settings** → **General** and click **Edit Checklist**. From the menu, click **Add Your First Task**. ### Add a Pre-Defined Task GrowthBook offers 5 pre-defined, common tasks: 1. Add a descriptive hypothesis for this experiment 2. Upload a screenshot for each variation of the experiment 3. Add a description for this experiment 4. Add this experiment to a project 5. Add at least 1 tag to this experiment GrowthBook evaluates the status of these tasks automatically, checking off the task when the requirement is met. ### Add a Custom Task To define a custom task, type the item in the text box and click **Add Task**. Add an optional URL to link to resources or documentation. :::tip Reorder tasks by dragging and dropping them into the desired order. ::: Click **Save** to enable the checklist on all experiments. ### Project-Level Checklists You can also create a project-level checklist. This checklist will override the organization-level checklist for all experiments in the project. To create a project-level checklist, go to **Settings** → **Projects** and select the project you want to create a checklist for. Here you can scroll to the `Experiment Settings` section and click `Create Checklist`. If you change your mind, and you'd instead just like to use your organization's default checklist, you can delete the project-level checklist by clicking the `Delete Checklist` button. --- ## Holdouts import CommercialFeature from '@site/src/components/CommercialFeature'; Holdout experiments (or simply "holdouts") measure the long-term impact of features by maintaining a control group that doesn't receive new functionality. While most users experience your latest features and improvements, a small percentage remains on the original version, providing a baseline to measure cumulative effects over time. Learn more about holdouts in our [Knowledge Base](/kb/experiments/holdouts). ## How Holdouts Work in GrowthBook In GrowthBook, units (e.g. users) in the holdout group are withheld from new features, experiments, and bandits. Units not in the holdout group will: participate in experiments, receive feature releases, and see all your latest changes. A subset of these units not in the holdout are used for measurement; these units are the same as the rest of the general population, we just select a sub-sample of units to serve as a comparison group for your holdout population. This approach measures the cumulative impact of all changes over time by comparing the general population to the holdout group. Under the hood, holdouts work by using prerequisites to divert a selected percentage of traffic away from features and experiments towards default values selected for your feature flags. ## Creating a Holdout ![Add Holdout](/images/holdouts-add-modal.png) 1. Navigate to **Experiments** → **Holdouts** and click **Add Holdout** 2. Configure your holdout settings: - **Name**: Choose a descriptive name (e.g., `q1-2025-product-holdout`) - **Holdout Size**: Defaults to 5% (meaning 5% of traffic is placed in the holdout). Note: We also use this % as the amount of traffic in the general population to use for measurement. For example, if you pick 5% for your holdout group, then you will have 5% in the holdout and 95% not in the holdout, but that 95% is split into 5% that are used for measurement and 90% that are not. While using the full 95% for measurement would increase the statistical power of your holdout, it does not represent good statistical value to do so. The statistical power of a test is mostly limited by the size of the smaller group, which in this case is the holdout group, and adding a ton of additional tracking events and ballooning the holdout sample for this (often) long running experiment will cause more events to fire and queries to run longer, often without a meaningful reduction in uncertainty in your test. Please reach out if you wish to implement a different size measurement population. - **Description**: Document the purpose and scope of your holdout - **Projects**: Select the projects in which the holdout should be active. The holdout will be added by default to new features, experiments, and bandits in this project and will only be available to add to these projects. - **Enabled Environments**: Holdouts are automatically added to any linked features in the selected environments (typically, only production). - **Assign Variation by Attribute**: Select the attribute to split traffic into your holdout, typically a user identifier. Note: when you add the holdout to future features or experiments, the holdout will automatically apply to any evaluation of the feature flag and will use the selected attribute to divert traffic to the holdout. - **Goal Metrics**: Select the key metrics for measuring cumulative impact. Note: We prevent adding metrics with conversion windows to a holdout. This is because traffic is exposed to the holdout whenever they hit an included feature or experiment. This can often be days or weeks apart from their exposure to a later feature or experiment, so conversion windows based around their first exposure event no longer make sense. Furthermore, the goal of a holdout is to measure long-range effects; so we allow you to add metrics without conversion windows or with lookback windows. 3. Click **Start Holdout** to begin ## Adding to a Holdout Once your holdout is created, it is selected by default in new feature and experiment creation flows for the selected projects. Whenever someone creates a new feature, experiment, or bandit in a project the holdout was assigned to, a new field for "Holdout" is added to the creation flow and it defaults to the created holdout. If there are multiple holdouts assigned to a project (not recommended), then the feature or experiment creator will have to select the correct holdout to use, as only one can be used at a time per feature or experiment. ### Adding Experiments to a Holdout ![Add Holdout to an experiment](/images/holdouts-experiment-with-holdout.png) Once created, you can use your holdout in new experiments: 1. When creating a new experiment, you'll see a **Holdout** field. Note: Experiments created from a feature that has a holdout applied will automatically inherit the holdout from that feature. 2. Your active holdout for the given project will selected in the dropdown by default 3. Complete the rest of your experiment setup normally The experiment will run as usual, with the holdout tracking long-term cumulative effects in the background. ## Holdout Lifecycle ### Active Period Throughout the active holdout period, you can use the holdout to measure cumulative impact on your metrics as new features and experiments are launched. However, it's often best to measure holdout impact for some set period of time after all new changes are frozen in place and the general population has had a chance to experience them together. For this reason, we make it easy to measure impacts over a set analysis period. During the active period (often a quarter), you'll: - Continue adding new experiments to the holdout - Launch features as normal - Let the holdout accumulate data on cumulative impact ### Analysis Period After your planned duration, transition to the analysis period: 1. **Stop adding new experiments**: The holdout enters analysis mode where no new experiments or features can be added but users are still held out from previously added experiments and features. 2. **Extended measurement**: Continue measuring for 2-4 weeks to capture delayed effects and reduce statistical noise 3. **Dynamic lookback window**: GrowthBook automatically applies a lookback window based on when the analysis window began, ensuring clean measurement During analysis, the system: - Continues to split traffic on features and experiments into your holdout and general population groups if the holdout was already applied. - Applies the appropriate lookback window to exclude results from before the analysis period - Uses the minimum of the metric's lookback window (if it exists) and the analysis lookback window for each metric ### Understanding Holdout Results Holdout results show the cumulative impact of all features and experiments included during the holdout period: - **Positive lift**: Your combined changes improved metrics beyond the baseline - **Negative or neutral lift**: Consider whether individual wins masked broader negative effects - **Metric-specific impacts**: Different metrics may show varying long-term effects The analysis compares your general population (who received all features and were experimented on) against the holdout control group (who remained on the baseline), revealing the true long-term value of your product changes. ## FAQ ### I don't see the holdout when creating my feature or experiment, where is it? Holdouts must be started and must be in the same project as the feature or experiment to be selected in the dropdown. ### Can I have more than one holdout on a feature or experiment? No, you can only have one holdout on a feature or experiment. ### Can I have more than one holdout active at once? Yes, but it is not recommended to have multiple holdouts active at once within a project. If there are more than one available at a time, when creating a new feature or experiment, the user will have to select the correct holdout to use. ### How can I add a Holdout to an existing experiment? Holdouts can only be added to existing experiments if they are (1) in a draft state, (2) no existing linked feature flags, and (3) no existing holdout on the experiment. You can use the three dot menu in the top right of the experiment to add a holdout to the experiment. If there is an existing linked feature flag but the experiment is still in a draft state, you can go to the feature and add the holdout directly to the feature first, and then you can add the holdout to the experiment. ### How can I add a Holdout to an existing feature? Holdouts can only be added to existing features if they have no rules set up in the feature and never have had a linked experiment. You can use the three dot menu in the top right of the feature to add a holdout to the feature. --- ## URL Redirect Testing import Pill from '@site/src/components/Pill'; import CommercialFeature from '@site/src/components/CommercialFeature'; # URL Redirect Testing URL Redirect tests are an alternative to using the Visual Editor and are ideal for testing big changes or complete page redesigns. ## How it Works You start by specifying an "Original URL". Users who visit this URL will be included in the experiment (assuming they meet all of the other targeting conditions). You then specify "Destination URLs" for each of your variations. You can also turn off redirects for some of your variations if you want to keep the user on the Original URL. When a user visits the Original URL and is included in the experiment, they will be assigned a variation. Then, an "Experiment Viewed" event will be sent to your data warehouse. After a short delay (default `100ms`) to allow for the tracking event to finish, the user will be redirected to the destination URL. ### Redirecting when only query params change Setting up tests where the redirect URL only varies from the original by the query parameters requires special consideration. You'll need to include the query params you plan to use in the original URL with no value entered in order for our URL targeting to work correctly (i.e. if you want to redirect a user to `http://www.growthbook.io/test?abc=123` from `http://www.growthbook.io/test`, your original URL will need to be `http://www.growthbook.io/test?abc`). ## Implementation URL Redirect tests require integrating one of our [SDKs](/lib) into your application. Currently, URL Redirect tests are only supported in our [Script Tag](/lib/script-tag), [Javascript](/lib/js), and [ReactJS](/lib/js) SDKs—as well as natively supported in all [Edge SDKs](/lib/edge/cloudflare). Verify URL Redirect tests are enabled for your SDK Connection: Go to **SDK Connections**, select your SDK connection, click **Edit**, and ensure that "Enable URL Redirect experiments" is toggled on. ### HTML Script Tag SDK The easiest option is to use our [Script Tag](/lib/script-tag) SDK. This involves adding a single `` tag to the HEAD of your website. This option fully works out-of-the-box with no configuration required. Like the JavaScript and React SDKs, the HTML Script tag supports setting a custom `navigate` function, `navigateDelay`, and `maxNavigateDelay`. See the [JavaScript and React section](#front-end-javascript-and-react-sdks) below for more information. #### Anti-flicker The Script Tag is the only front-end SDK that natively provides an anti-flicker mechanism (off by default). - `antiFlicker`: When the SDK loads, it displays a blank screen that persists until either a URL redirect finishes _or_ no redirect is triggered. This helps to minimize any "flicker" a user might see while the page navigates. Defaults to `false`. - `antiFlickerTimeout`: The maximum number of milliseconds that the anti-flicker blank screen will be shown. Defaults to `3500` (3.5 seconds). **Strategies for Improving Anti-flicker** A small flicker will remain between page and SDK load (which usually happens asynchronously). To completely eliminate all flicker between initial page load and redirection, follow these steps: 1. Add this code to the page ``: ```html .gb-anti-flicker { opacity: 0 !important; pointer-events: none; } ``` 2. Add the `gb-anti-flicker` class to the page ``: ```html ``` 3. Optionally, add a script to the `` to remove your anti-flicker screen after 3 seconds in case the SDK never successfully loads: ```html window.setTimeout(() => { document.documentElement.classList.remove("gb-anti-flicker"); }, 3000); ``` ### Front-End JavaScript and React SDKs A more advanced integration involves using our [Javascript](/lib/js) or [ReactJS](/lib/js) client-side SDKs. Besides the standard implementation described in the SDK docs, there are additional settings to control redirect behavior. - `navigate`: A callback function to perform the redirect. Defaults to `(url) => window.location.replace(url)`. - `navigateDelay`: The number of milliseconds to wait before redirecting. Use this delay to give time for your analytics tracking callback to finish. If your tracking callback is properly implemented as async, then you should set this value as low as possible. Defaults to `100` (0.1 seconds). - `maxNavigateDelay`: The maximum number of milliseconds that navigation will be delayed while any async tracking callback is still in flight. If the tracking callback exceeds this value, `navigate` will be fired. Missed tracking calls are generally the largest contributor to [Sample Ratio Mismatch (SRM)](/using/experimenting#sample-ratio-mismatch-srm). Defaults to `1000` (1 second). _Deprecated:_ - `antiFlicker`: Only available in the Script Tag. - `antiFlickerTimeout`: Only available in the Script Tag. #### Single Page Apps (SPAs) If you have a SPA, it's recommended to use your own `navigate` function to avoid full-page redirects. In this case, also set `navigateDelay` to `0`. Here is an example in Next.js: ```jsx import router from "next/router"; const gb = new GrowthBook({ navigate: (url) => router.replace(url), navigateDelay: 0, maxNavigateDelay: 0 // only needed if your tracking callback is async // ... other settings }); ``` It's also necessary to update the URL in the GrowthBook instance on every client-side navigation. For example: ```jsx router.events.on("routeChangeComplete", (url) => { gb.setURL(url); }) ``` ### Edge SDKs The best practice for running URL redirect tests is to implement them on the backend or edge server (CDN). In a front-end environment, failing to complete the tracking callback before the navigation triggers can lead to substantial [Sample Ratio Mismatch (SRM)](/using/experimenting#sample-ratio-mismatch-srm) errors and could invalidate your experiment results. Similarly, flickering can cause a degraded user experience, which may end up skewing the experiment results in favor of the control (no redirect). We provide native support for edge redirect tests without screen flickering or tracking delays. Learn more about our [Edge SDKs](/lib/edge/cloudflare#turnkey-edge-app). ### Node.js beta It is possible to use our [Javascript](/lib/js) SDK to perform redirects on the back-end. You must implement your own `navigate` function. ## Sample Ratio Mismatch (SRM) Warnings on URL Redirect Tests [Sample Ratio Mismatch (SRM) warnings](/using/experimenting#sample-ratio-mismatch-srm) happen when the actual traffic split in an experiment is significantly different from the traffic split configured for the experiment. For instance, if a 50/50 traffic split was configured for an experiment but a significantly different traffic split is observed, like a 46/54 split, an SRM warning will be triggered. GrowthBook only shows SRM warnings if the p-value is less than 0.001, which means it's extremely unlikely to occur by chance. This p-value threshold is customizable in **Settings** → **General** → **Experiment Settings** → **Experiment Health Settings**. ### Why am I seeing SRM warnings on my URL Redirect test? SRM warnings are a common problem when running front-end URL redirect tests, regardless of what experimentation platform is used. When using GrowthBook's [JavaScript](/lib/js), [React](/lib/react), [Vue.js](/lib/vue), or [Script Tag](/lib/script-tag) SDKs to run URL Redirect tests, SRM warnings may occur. The issue is especially pronounced while also using [Google Tag Manager (GTM)](/guide/google-tag-manager-and-growthbook). The inherent latency of the [trackingCallback](/lib/js#experimentation-ab-testing) sometimes causes the URL Redirect to take place before the `trackingCallback` can complete, which means a disproportionately larger number of exposures are tracked for the control URL versus the redirect URL. This triggers an SRM warning. ### What can I do to reduce or eliminate SRM warnings? 1. The best way to eliminate a Sample Ratio Mismatch with URL Redirect tests is to use one of our **Edge Worker SDKs** instead of a front-end SDK. See our Edge SDK documentation for [Cloudflare Workers](/lib/edge/cloudflare), [Fastly Compute](/lib/edge/fastly), and [AWS Lambda@Edge](/lib/edge/lambda). 2. If you opt to use a front-end SDK (JS, React, or Vue), you can still greatly reduce or nearly eliminate SRM by implementing an **async tracking callback**. If your event tracking vendor supports async callbacks, then simply ensure that the `trackingCallback` that you pass into the GrowthBook SDK constructor awaits a successful network request for the tracked event. If you are using our HTML Script Tag, this is automatically handled for _GA4_ and _GTM_ tracking calls. :::note Async tracking callbacks GrowthBook JavaScript and React SDKs introduced **async tracking callbacks** in version `1.2.0`. ::: 3. If you are unable to use an async tracking callback, you can try increasing the `navigateDelay` timeout passed into the GrowthBook SDK constructor. This defers the URL redirection until after a minimum time, increasing the likelihood that your tracking callback is successfully dispatched before page redirection. The default delay is 1000 ms (1 second), however some network conditions or SDK implementations do better with a navigate delay of 2 to 3 seconds. _Segment_ recommends a 300 ms delay; if you are using our HTML Script Tag with _Segment_, this is automatically set to 300 ms. --- ## Using a CDN # Using a CDN You can add a CDN in between your GrowthBook SDKs and your GrowthBook API Server. This will act as a caching layer when serving feature flags and enable virtually limitless scaling with very low maintenance costs. All major CDNs are supported including Cloudflare, Fastly, Akamai, CloudFront, Azure Front Door, and Google Cloud CDN. Using a CDN is a quick and easy solution to scaling. If you want more power and control, we recommend deploying our [GrowthBook Proxy Server](/self-host/proxy) instead. :::note GrowthBook Cloud CDN GrowthBook Cloud provides its own CDN for free for low/medium traffic apps (less than 1 million requests per month). If you plan to go above that limit, this guide will work for you as well. Just use `https://cdn.growthbook.io` as your Origin instead of a self-hosted GrowthBook API server. ::: ## How it Works 1. Add a CDN that sits in front of your GrowthBook API server 2. Configure the CDN to cache requests to `/api/features/*` (and deny all other routes) 3. Point your SDKs to your CDN's domain instead of your GrowthBook API Server 4. (optional) Configure a Webhook to invalidate the CDN cache when features change within GrowthBook ## Configuring your CDN ### Caching Rules The `/api/features/*` route includes a default `Cache-control` header that should work out-of-the-box for most CDNs: ``` public, max-age=30, stale-while-revalidate=3600, stale-if-error=36000 ``` This will serve all requests from cache and refresh in the background every 30 seconds. You can override these settings in your CDN if desired. **Important**: Make sure to block all other routes besides `/api/features/*`. Doing this will greatly increase security by limiting the potential attack surface. ### Health Checks The GrowthBook API server exposes a `/healthcheck` route. You can optionally use this in your CDN so it can tell if your origin server goes down. ## SDK Configuration Most of our SDKs accept an `apiHost` setting. Simply change this from your GrowthBook API server to the new CDN domain. ## Purging / Invalidation When you make changes withing GrowthBook, the CDN will not pick up the changes until it refreshes its cache (default 30 seconds). For most apps, this short delay is perfectly fine. If you want faster rollouts, you can have GrowthBook tell your CDN to refresh as soon as changes are made. ### Fastly If you use Fastly as your CDN and are self-hosting GrowthBook, you can set the following environment variables to automatically purge URLs: - `FASTLY_SERVICE_ID` - `FASTLY_API_TOKEN` ### HTTP PURGE Requests If your CDN supports invalidating URLs with simple HTTP PURGE requests, you can add an [SDK Webhook](/app/webhooks/sdk-webhooks) in GrowthBook and set the method as PURGE with no payload. ### Other If the methods above are not available for your CDN, you can configure a custom [SDK Webhook](/app/webhooks/sdk-webhooks). We let you customize the HTTP method and headers, but if you need more control, you may need to set up a custom script that sits between the webhook and the CDN. --- ## Config.yml # Config.yml In order to use GrowthBook for experimentation, you need to connect to a data source. There are two main ways to configure GrowthBook: via the UI, or via a `config.yml` file. The default way to define these is by filling out forms in the GrowthBook UI, which persists them to MongoDB. The other option is to create a `config.yml` file, which allows for version control and easier management of your configuration. In the Docker container, this file must be placed at `/usr/local/src/app/config/config.yml`. Below is an example file: ```yml datasources: warehouse: type: postgres name: Main Warehouse # Connection params (different for each type of data source) params: host: localhost port: 5432 user: root password: ${POSTGRES_PW} # use env for secrets database: growthbook # How to query the data (same for all SQL sources) settings: userIdTypes: - userIdType: user_id description: Logged-in user id - userIdType: anonymous_id description: Anonymous visitor id queries: exposure: - id: user_id name: Logged-in user experiments userIdType: user_id query: > SELECT user_id, received_at as timestamp, experiment_id, variation_id, context_location_country as country FROM experiment_viewed dimensions: - country identityJoins: - ids: ["user_id", "anonymous_id"] query: SELECT user_id, anonymous_id FROM identifies ``` ### Data Source Connection Params The contents of the `params` field for a data source depends on the type. As seen in the example above, you can use environment variable interpolation for secrets (e.g. `${POSTGRES_PW}`). #### Redshift, ClickHouse, Postgres, and Mysql (or MariaDB) ```yml type: postgres # or "redshift" or "mysql" or "clickhouse" params: host: localhost port: 5432 user: root password: password database: growthbook ``` Redshift and Postgres also support optional params to force an SSL connection: ```yml type: postgres params: ... ssl: true # Omit the below fields to use the default trusted CA from Mozilla caCert: "-----BEGIN CERTIFICATE-----\n..." clientCert: "-----BEGIN CERTIFICATE-----\n..." clientKey: "-----BEGIN CERTIFICATE-----\n..." ``` #### Snowflake ```yml type: snowflake params: account: abc123.us-east-1 username: user password: password database: GROWTHBOOK schema: PUBLIC role: SYSADMIN warehouse: COMPUTE_WH ``` #### BigQuery You must first create a Service Account in Google with the following roles: - Data Viewer - Metadata Viewer - Job User If you want GrowthBook to auto-discover credentials from environment variables or GCP metadata, use the following: ```yml type: bigquery params: authType: auto ``` If you prefer to pass in credentials directly, you can use this format instead: ```yml type: bigquery params: projectId: my-project clientEmail: growthbook@my-project.iam.gserviceaccount.com privateKey: -----BEGIN PRIVATE KEY-----\nABC123\n-----END PRIVATE KEY-----\n ``` #### Presto and TrinoDB ```yml type: presto params: engine: presto # or "trino" host: localhost port: 8080 username: user password: password catalog: growthbook schema: growthbook ``` #### Databricks ```yml type: databricks params: host: dbc-123-abc.cloud.databricks.com port: 443 path: /sql/1.0/warehouses/abc123 token: dapi123abc ``` #### AWS Athena If you want GrowthBook to auto-discover credentials from environment variables or instance metadata, use the following format: ```yml type: athena params: authType: auto region: us-east-1 database: growthbook bucketUri: aws-athena-query-results-growthbook workGroup: primary ``` If you prefer to specify access key and secret directly instead, use the following format: ```yml type: athena params: accessKeyId: AKIA123 secretAccessKey: AB+cdef123 region: us-east-1 database: growthbook bucketUri: aws-athena-query-results-growthbook workGroup: primary ``` ### Data Source Settings The settings tell GrowthBook how to query your data. There are a couple queries you need to define plus an optional Python script to run queries from inside a Jupyter notebook: ```yml type: postgres params: ... settings: # The different types of supported identifiers userIdTypes: - userIdType: user_id description: Logged-in user id - userIdType: anonymous_id description: Anonymous visitor id queries: # These queries returns experiment variation assignment info # One row every time a user was put into an experiment exposure: - id: user_id name: Logged-in user experiments userIdType: user_id query: > SELECT user_id, received_at as timestamp, experiment_id, variation_id, context_location_country as country FROM experiment_viewed # List additional columns you selected in your experimentsQuery # Can use these to drill down into experiment results dimensions: - country # These optional queries map between different types of identifiers identityJoins: - ids: ["user_id", "anonymous_id"] query: SELECT user_id, anonymous_id FROM identifies # Used when exporting experiment results to a Jupyter notebook # Define a `runQuery(sql)` function that returns a pandas data frame notebookRunQuery: > import os import psycopg2 import pandas as pd from sqlalchemy import create_engine, text # Use environment variables or similar for passwords! password = os.getenv('POSTGRES_PW') connStr = f'postgresql+psycopg2://user:{password}@localhost' dbConnection = create_engine(connStr).connect(); def runQuery(sql): return pd.read_sql(text(sql), dbConnection) ``` ### Organization Settings Some organization settings can also be controlled from `config.yml`. Below are all of the currently supported settings: ```yml organization: settings: # Minimum experiment length (in days) when importing past experiments. Default `6` pastExperimentsMinLength: 3 # Number of days of historical data to use when analyzing metrics # (must be between 1 and 400, default `90`) metricAnalysisDays: 90 # The min percent of users exposed to multiple variations in an # experiment before we start warning you (between 0 and 1, defaults to `0.01`) multipleExposureMinPercent: 0.01 # Whether Regression Adjustment (CUPED) should be on or off by default how many # days to use. Can be overridden in your metric definitions if you wish. regressionAdjustmentEnabled: true regressionAdjustmentDays: 14 # When we should auto-update experiment results updateSchedule: type: stale hours: 6 ``` The `updateSchedule` setting has 3 types of values: - Never update automatically ```yml updateSchedule: type: never ``` - Update if data is X hours stale ```yml updateSchedule: type: stale hours: 6 ``` - Update on a fixed Cron schedule ```yml updateSchedule: type: cron cron: "0 */6 * * *" ``` ### Unit Dimensions Unit Dimensions let you join additional tables in order to drill down into your experiment results. Dimensions only have 4 properties: name, datasource, userIdType, and SQL. The SQL query must return two columns: the identifier type and `value`. Example: ```yml name: Country # Must match one of the datasources defined in config.yml datasource: warehouse userIdType: user_id sql: SELECT user_id, country as value FROM users ``` By default, when using `config.yml`, it's not possible to create dimensions via the GrowthBook UI. Everything must be done directly in the `config.yml` file. There is an optional environment variable you can specify to change this behavior: ```sh ALLOW_CREATE_DIMENSIONS=true ``` This will let you create new dimensions via the UI. Dimensions defined in `config.yml` will be marked as "Official" and not editable, while ones defined via the UI will be editable. ### Metrics (Deprecated) Legacy metrics can be defined in the `config.yml` file as well, although this behavior is deprecated. If you want to version control metrics, the recommended approach is to integrate your version control system with our API instead. [See a guide here for doing this with GitHub](/integrations/github-metrics). Below is an example of all the possible settings with comments for legacy metrics defined in `config.yml`: ```yml name: Revenue per User # Required. The data distribution and unit type: revenue # or "binomial" or "count" or "duration" # Required. Must match one of the datasources defined in config.yml datasource: warehouse # Description supports full markdown description: This metric is **super** important # For inverse metrics, the goal is to DECREASE the value (e.g. "page load time") inverse: false # When ignoring nulls, only users who convert are included in the denominator # Setting to true here would change from "Revenue per User" to "Average Order Value" ignoreNulls: false # Which identifier types are supported for this metric userIdTypes: - user # Any user with a higher metric amount will be capped at this value # In this case, if someone bought a $10,000 order, it would only be counted as $100 # Note: you can also specify `type: percentile` and a `value` between 0 and 1 # for percentile based capping cappingSettings: type: absolute value: 100 # Control the date window for your metrics windowSettings: type: conversion # Ignore all conversions within the first X hours of being put into an experiment. delayHours: 0 # After the conversion delay (if any), wait this many hours for a conversion event. windowValue: 72 windowUnit: hours # Min number of conversions for an experiment variation before we reveal results minSampleSize: 150 # The "suspicious" threshold. If the percent change for a variation is above this, # we hide the result and label it as suspicious. # Default 0.5 = 50% change maxPercentChange: 0.50 # The minimum change required for a result to considered a win or loss. If the percent # change for a variation is below this threshold, we will consider an otherwise conclusive # test a draw. # Default 0.005 = 0.5% change minPercentChange: 0.005 # Overrides for Regression Adjustment (CUPED) at the metric level. To enforce these fields # you must set regressionAdjustmentOverride to true. # Leave the settings out of your config file to accept your organization level settings; or # set regressionAdjustmentOverride to false. regressionAdjustmentOverride: true regressionAdjustmentEnabled: true regressionAdjustmentDays: 14 # Arbitrary tags used to group related metrics tags: - revenue - core ``` In addition to all of those settings, you also need to tell GrowthBook how to query the metric with SQL. Depending on the other settings, the columns you need to select may differ slightly: - `timestamp` - always required - `value` - required unless type is set to "binomial" Plus, you need to select a column for each identifier type the metric supports. A full example: ```yml type: duration userIdTypes: - user_id - anonymous_id sql: > SELECT created_at as timestamp, user_id, anonymous_id, duration as value FROM requests ``` And a simple binomial metric that only supports logged-in users: ```yml type: binomial userIdTypes: - user sql: SELECT user_id, timestamp FROM orders ``` By default, if a user has more than 1 non-binomial metric row during an experiment, we sum the values together. You can override this behavior with the `aggregation` setting: ```yml type: duration userIdTypes: - user_id sql: > SELECT created_at as timestamp, user_id, duration as value FROM requests aggregation: MAX(value) # use MAX instead of the default SUM ``` #### Allow Creating Metrics in the UI By default, when using `config.yml`, it's not possible to create legacy metrics via the GrowthBook UI. Everything must be done directly in the `config.yml` file. There is an optional environment variable you can specify to change this behavior: ```sh ALLOW_CREATE_METRICS=true ``` This will let you create new metrics via the UI. Metrics defined in `config.yml` will be marked as "Official" and not editable, while ones defined via the UI will be editable. --- ## Environment Variables # Environment Variables The default configuration in GrowthBook is optimized for trying things out quickly on a local dev machine. Beyond that, you can customize behavior with Environment Variables. ## Domains and Ports This is used to generate links to GrowthBook and enable CORS access to the API. - **APP_ORIGIN** - default http://localhost:3000 - **API_HOST** - default http://localhost:3100 If you want to run GrowthBook on a different host than localhost, you need to add the above environment variables to your `docker-compose.yml` file. ```yml growthbook: ... environment: - APP_ORIGIN=http://:3000 - API_HOST=http://:3100 ``` :::info Unique ports on the same domain The back-end and the front-end **must run on unique ports if on the same domain**. The front-end has its own set of `/api` routes so currently we do not support running both the API and the front-end on the same port on the same domain. ::: :::info Important In order for authentication cookies to work correctly, both the app and api domains must be considered the "same site". They can be on different subdomains or ports, but the root domain must be the same. Works: gb.example.com and{" "} gb-api.example.com Breaks: gb.frontend.com and{" "} gb.backend.com ::: If you need to change the ports on your local dev environment (e.g. if `3000` is already being used), you need to update the above variables AND change the port mapping in `docker-compose.yml`: ```yml growthbook: ports: - "4000:3000" # example: use 4000 instead of 3000 for the app - "4100:3100" # example: use 4100 instead of 3100 for the api ... environment: - APP_ORIGIN=http://:4000 - API_HOST=http://:4100 ``` If you for some reason have a **PORT** environment variable set, you need to set the **BACKEND_PORT** variable to `3100` to avoid port conflicts. ### Production Settings - **NODE_ENV** - Set to "production" to turn on additional optimizations and API request logging - **MONGODB_URI** - The full MongoDB connection string. Alternatively you can specify the following environment variables from which we will compose the full connection string like so `mongodb://${MONGODB_USERNAME}:${MONGODB_PASSWORD}@${MONGODB_HOSTNAME}:${MONGODB_PORT}/${MONGODB_DBNAME}${MONGODB_EXTRA_ARGS}`: - **MONGODB_USERNAME** - Username for MongoDB - **MONGODB_PASSWORD** - Password for MongoDB - **MONGODB_HOST** - Host name of MongoDB excluding port, e.g. `some.host.com` - **MONGODB_PORT** - Port to use for MongoDB (defaults to `27017`) - **MONGODB_DBNAME** - Database name of growthbook database (defaults to `growthbook`) - **MONGODB_EXTRA_ARGS** - Optional set of extra arguments for MongoDB connection string (defaults to `?authSource=admin`) - **JWT_SECRET** - Auth signing key (use a long random string) - **ENCRYPTION_KEY** - Data source credential encryption key (use a long random string) If you change the `ENCRYPTION_KEY`, you will need to migrate any existing data sources using the following script: ```bash # If you didn't have an ENCRYPTION_KEY before, leave OLD_KEY blank below docker compose run --workdir /usr/local/src/app/packages/back-end \ growthbook node dist/scripts/migrate-encryption-key.js OLD_KEY ``` :::info Important When using GrowthBook in production, it is important to change the NODE_ENV to to "production" and change the JWT_SECRET to a random string. Using the production node environment and the default JWT_SECRET will throw an error. ::: ## Email SMTP Settings This is required in order to send experiment alerts, team member invites, and reset password emails. - **EMAIL_ENABLED** ("true" or "false") - **EMAIL_HOST** - **EMAIL_PORT** - **EMAIL_HOST_USER** - **EMAIL_HOST_PASSWORD** - **EMAIL_FROM** ## Google OAuth Settings Only required if using Google Analytics as a data source - **GOOGLE_OAUTH_CLIENT_ID** - **GOOGLE_OAUTH_CLIENT_SECRET** ## File Uploads The **UPLOAD_METHOD** environment variable controls where to store uploaded files and screenshots. The supported values are `local`, `s3`, and `google-cloud`. ### local This is the default value. Uploads are stored in the GrowthBook docker container at `/usr/local/src/app/packages/back-end/uploads`. In production, you should mount a volume here to persist uploads across container restarts. ### s3 Store uploads in an AWS S3 bucket. - **S3_BUCKET** - **S3_REGION** (defaults to `us-east-1`) - **S3_DOMAIN** (defaults to `https://${S3_BUCKET}.s3.amazonaws.com/`) - **AWS_ACCESS_KEY_ID** (not required when deployed to AWS with an instance role) - **AWS_SECRET_ACCESS_KEY** (not required when deployed to AWS with an instance role) - **AWS_ASSUME_ROLE** (optional role to assume, that has `s3:PutObject` and `s3:GetObject` to the bucket) ### google-cloud Store uploads in a Google Cloud Storage bucket. - **GCS_BUCKET_NAME** - **GCS_DOMAIN** (defaults to `https://storage.googleapis.com/${GCS_BUCKET_NAME}/`) - **GOOGLE_APPLICATION_CREDENTIALS** (not required when deployed to GCP with an instance service account) ## AI Provider API Keys GrowthBook supports multiple AI providers for features like generating experiment descriptions, checking hypothesis validity, and detecting similar experiments. You can configure one or more of the following API keys: - **OPENAI_API_KEY** - OpenAI API key for GPT models - **ANTHROPIC_API_KEY** - Anthropic API key for Claude models - **XAI_API_KEY** - xAI API key for Grok models - **MISTRAL_API_KEY** - Mistral AI API key - **GOOGLE_AI_API_KEY** - Google AI API key for Gemini models You only need to set the API keys for the providers you want to use. Once configured, you can select your preferred model in the AI settings. See more on the [AI Integrations page](/integrations/ai). ## Enterprise Settings Some features in self-hosted GrowthBook are only available with a commercial license key. You can get a trial Enterprise key from your self-hosted version of GrowthBook, or reach out to [sales@growthbook.io](mailto:sales@growthbook.io) to learn more. ### License Keys If you have a license key, you can activate it in GrowthBook two different ways: 1. Navigate to **Settings** → **General** and look for the License section of the page. There you can input or edit your license key string. (Recommended) 2. Alternately, set the environment variable `LICENSE_KEY` to your license key string. If using Docker, go to _docker-compose.yml_ and add the variable to the _growthbook_ → _environment_ section. :::note License verification If you are running your application behind a firewall you will need to whitelist `75.2.109.47` to be able to have your license verified. If this is an issue, contact sales@growthbook.io for an air-gapped license. ::: The GrowthBook app will check with the license server daily to get the latest license information. When doing so, it will send the following data to our license server: - **A hash of each user's email** - this is used to deduplicate users across multiple installations (dev, prod, etc.) in order to figure out how many seats are being used. - **The git sha and date of the installation** - this is used to determine when GrowthBook can delete deprecated features, determine who to send upgrade notices to, provides helpful information for debugging, etc. - **A list of which data source types are being used, a list of which event trackers are being used, and a list of SDK languages being used** - these help GrowthBook know how much usage the various data sources, event trackers, and sdk languages are getting so we can prioritize work. If you recently upgraded your license and do not want to wait a day to get the new license information you can go to **Settings** → **General** and look for the License section of the page and click the Refresh button. If you have a multi-org installation and are a super admin you must instead navigate to the **Settings** → **Admin** page, find the license section and click the refresh button there. ### Enterprise SSO To enable SSO on your self-hosted GrowthBook instance, you will need an active license key (see above section), and then you may add the SSO settings for your provider. You can find the instructions on how to setup SSO for GrowthBook with most common providers on our [SSO instructions page](/sso). The JSON object with the settings will need to be JSON encoded and then set to the environment variable `SSO_CONFIG`. ### Multi-org If your company has two or more largely independent products (e.g. Google has Search and Google Docs), you can set up multiple [organizations](/using/growthbook-best-practices#organizations) per product. The following env vars control multi-org settings: - **IS_MULTI_ORG** - This defaults to `false`. Set this to true if you want to enable multiple organizations. - **ALLOW_SELF_ORG_CREATION** - This defaults to `false`. Set this to true to allow any users to create new organizations. - **SHOW_MULTI_ORG_SELF_SELECTOR** - This defaults to `true`. It allows all users to see a list of all organizations they can join. Upon selecting an organization, users will either automatically join it (if using SSO and [self-join](/account/user-permissions#self-registering-and-automatic-approvals) is enabled for that organization) or require an admin's approval. ## Observability (OpenTelemetry) The GrowthBook API is instrumented with OpenTelemetry to publish observability metrics, traces, and logs. To enable, you must set the environment variable `TRACING_PROVIDER=opentelemetry`. The standard [OTEL\_\* Environment Variables](https://opentelemetry.io/docs/concepts/sdk-configuration/) are supported, such as `OTEL_SERVICE_NAME` and `OTEL_EXPORTER_OTLP_ENDPOINT`. Logs collection is supported but disabled by default. You can set the `GROWTHBOOK_OTEL_ENABLE_LOGS_COLLECTION` environment variable to enable it. ## Other Settings - **EXPERIMENT_REFRESH_FREQUENCY** - Default update schedule for experiment results. Update when stale for X hours (default `6`). - **QUERY_CACHE_TTL_MINS** - How long (in minutes) to cache and re-use SQL query results (default `60`) - **DEFAULT_CONVERSION_WINDOW_HOURS** - How many hours after being put into an experiment does a user have to convert. Can be overridden on a per-metric basis. (default `72`) - **DISABLE_TELEMETRY** - We collect usage telemetry to help us improve GrowthBook. This includes a hashed organization ID, your user ID, IP address and derived geolocation, user agent, and app version. Set to "true" to disable. - **STORE_SEGMENTS_IN_MONGO** - If using the config.yml file, set to `true` if you want to store segments in Mongo. This is also useful if you have existing segments stored in Mongo that you need to access. When set, GrowthBook will ignore segments in the config.yml file and only use Segments stored in Mongo. - **CDN_HOST** - When set, this will update the implementation instructions within GrowthBook to override the `API_HOST` in cases where a CDN is used. - **EXPRESS_TRUST_PROXY_OPTS** - Express' [trust proxy](https://expressjs.com/en/5x/api.html#trust.proxy.options.table) setting value. Supports boolean (true/false), string values, and integer values for trusting the _n_th hop from the front-facing proxy server as the client. Leave empty or specify `false` to use Express' default behavior. If you are running GrowthBook behind a proxy or load balancer, this is required to track the correct user IP for audit log events. - **ROBOTS_TXT_PATH** - Define an absolute path to a robots.txt file that will be served on the `/robots.txt` path of your backend domain. If this env var is not defined, the app will look for a robots.txt file in the parent folder of the built `server.js` (which should be the `packages/back-end` directory in the default configuration) - **DISABLE_API_ROOT_PATH** - We include a default path in the API server to expose certain settings. Set to "true" to withhold that information. The Visual Editor however relies on the information in this path, so disabling it will prevent some of its features from working correctly. --- ## FerretDB # FerretDB FerretDB is a MongoDB-compatible database that is open source and free to use. It serves as a drop-in replacement for MongoDB, converting MongoDB wire protocol queries to SQL and using PostgreSQL as its backend storage engine. For organizations that require a 100% open-source stack, FerretDB provides a truly open-source MongoDB-compatible database alternative—free from SSPL licensing restrictions—allowing you to run GrowthBook without any proprietary database dependencies. ## Install FerretDB Follow the the [FerretDB installation guide](https://docs.ferretdb.io/installation/ferretdb/docker/) to install FerretDB. Make sure your FerretDB instance is running and accessible before proceeding. ## Start GrowthBook with FerretDB ```bash docker run -d \ --name growthbook \ --network ferretdb \ -p 3000:3000 \ -p 3100:3100 \ -v growthbook_uploads:/usr/local/src/app/packages/back-end/uploads \ -e MONGODB_URI="mongodb://username:password@:27017/growthbook" \ growthbook/growthbook:latest ``` Replace `username`, `password`, and `` with your FerretDB connection details. :::note Container network access Ensure that the FerretDB instance is accessible to the GrowthBook container. If both are running in Docker, make sure they're on the same network. ::: ## Launch GrowthBook That's it! GrowthBook is now running with FerretDB. Access it at `http://localhost:3000`. GrowthBook will automatically create the necessary collections in FerretDB to manage: - User authentication and permissions - Feature flags and their configurations - Metrics and experiments - Organization settings and metadata ## Verify the Connection After setting up GrowthBook, you can verify that data is being stored correctly in FerretDB by: - Creating a feature flag or experiment in GrowthBook - Connecting to your FerretDB instance using a MongoDB client - Checking the growthbook database for collections like experiments, features, and users ## Additional Resources - [FerretDB Documentation](https://docs.ferretdb.io/) - [FerretDB and GrowthBook Integration Blog Post](https://blog.ferretdb.io/enabling-feature-flags-ab-testing-growthbook-ferretdb/) - [FerretDB GitHub Repository](https://github.com/FerretDB/FerretDB) - [GrowthBook Self-Hosting Docs](/self-host) --- ## Overview # Self Hosting GrowthBook GrowthBook consists of a NextJS front-end, an ExpressJS API, and a Python stats engine. Everything is bundled together in a single [Docker Image](https://hub.docker.com/r/growthbook/growthbook). In addition to the app itself, you will also need a MongoDB instance (or MongoDB compatible database) to store login credentials, cached experiment results, and metadata. :::tip Don't want to install or host the app yourself? GrowthBook Cloud is a fully managed version that's free to get started with. ::: ## Quick Start You can use **docker compose** to get started quickly: ```yml # docker-compose.yml services: mongo: image: "mongo:latest" environment: - MONGO_INITDB_ROOT_USERNAME=root - MONGO_INITDB_ROOT_PASSWORD=password volumes: - mongodata:/data/db growthbook: image: "growthbook/growthbook:latest" ports: - "3000:3000" - "3100:3100" depends_on: - mongo environment: - MONGODB_URI=mongodb://root:password@mongo:27017/growthbook?authSource=admin volumes: - uploads:/usr/local/src/app/packages/back-end/uploads volumes: uploads: mongodata: ``` Then, just run `docker compose up -d` to start everything and view the app at [http://localhost:3000](http://localhost:3000) :::caution Use of mongo image The use of the mongo image within the docker-compose.yml is meant to quickly get a dev or staging environment up and running. For production you may want to use a more scalable and stable solution (ie. AWS DocumentDB, Google Cloud MongoDB Atlas, Azure Cosmos DB for Mongo, etc.) ::: We also offer an official Helm chart for deploying GrowthBook on Kubernetes. See the [Kubernetes Self-Hosting Guide](/self-host/kubernetes) for more details. ## Production We have a dedicated guide for [Self-Hosting GrowthBook in Production](/self-host/production) that covers best practices for security, scaling, performance, and more. ## Docker Tags The `latest` tag is updated with every commit to the main branch. We run this tag in GrowthBook Cloud, so it's generally safe for production, but may occasionally have minor bugs. We periodically create stable release tags (e.g. `4.1.0`) if you prefer to have more control over updates. Check out the [Releases page on GitHub](https://github.com/growthbook/growthbook/releases) for a list of releases with detailed release notes. Lastly, if you need to reference the image for a specific git commit for any reason, you can use the git shorthash tag (e.g. `git-41278e9`). ## Hardened (non-root, shell-less) image The GrowthBook runtime image is a distroless [Docker Hardened Image](https://www.docker.com/products/hardened-images): it runs as a **non-root user (uid 1000)** and contains **no shell or package manager**. This removes a recurring class of OS-package CVEs. If you pull the image directly (`docker run`, Docker Compose, or a pinned `latest`/git-sha tag), there are two things to know. ### Local file uploads must be writable by uid 1000 If you use `UPLOAD_METHOD=local`, a volume created by an older (root) GrowthBook image contains root-owned files that the non-root process can't write to. Newly created volumes work automatically; an **existing** volume needs a one-time ownership fix: ```bash # Docker named volume (e.g. the docker-compose "uploads" volume): docker run --rm -v :/data busybox chown -R 1000:1000 /data # Host bind-mount: sudo chown -R 1000:1000 /path/to/uploads ``` Alternatively, keep running as root by setting `user: "0:0"` on the service, or switch to `s3`/`google-cloud` storage (no local volume needed). On Kubernetes, the Helm chart sets `fsGroup: 1000`, which handles this for you. ### Debugging without a shell `docker exec sh` no longer works. To inspect a running container, attach a throwaway debug container that shares its process namespace: ```bash docker run --rm -it --pid=container: --cap-add=SYS_PTRACE busybox sh # the running container's filesystem is under /proc/1/root/, e.g.: # ls /proc/1/root/usr/local/src/app ``` `--cap-add=SYS_PTRACE` is required to read the target process's filesystem. Any image with a shell works (`busybox`, `debian`); the matching `dhi.io/node:-debian12-dev` image gives you the same libraries for running the app's own binaries. On Kubernetes, use `kubectl debug -it --image=busybox --target=`. One-off admin scripts (e.g. encryption-key migration) are run via `node` directly — see [Environment Variables](/self-host/env). --- ## Deploying with Kubernetes import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Deploying GrowthBook with Kubernetes :::tip Deploying GrowthBook with Kubernetes Our official Helm chart to deploy GrowthBook on Kubernetes is distributed as an OCI artifact via [ghcr.io](https://ghcr.io/growthbook/charts/growthbook): - For production and scalable cloud-native deployments, use GCP (GKE) or AWS (EKS). - For local testing and development, use Minikube. ::: ## Minimal Prerequisites Before you apply the example values files, make sure you complete these minimal prerequisites for your environment: **Kubernetes Cluster Access** Ensure you have access to a running Kubernetes cluster ([Minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download), [GKE](https://cloud.google.com/kubernetes-engine?hl=en), or [EKS](https://aws.amazon.com/eks/)) with permissions to deploy workloads and required controllers (such as load balancer controllers) installed. You'll need `kubectl` and the [GrowthBook Helm chart](https://artifacthub.io/packages/helm/growthbook/growthbook). **Authentication Secrets** Generate and Store JWT and encryption secrets for backend authentication and encryption: ```bash openssl rand -hex 32 # Run twice, once for each secret ``` Store them as Kubernetes secrets in the same namespace as the one where the chart will be installed (replace the values with your generated secrets): ```bash kubectl create secret generic jwt-secret \ --from-literal=jwt-secret=YOUR_JWT_SECRET kubectl create secret generic encryption-key \ --from-literal=encryption-key=YOUR_ENCRYPTION_KEY ``` Reference these secrets in your environment specific values file. **DNS Resolution for Testing** For quick testing without configuring DNS, you can use public DNS services that automatically resolve to your cluster's ingress IP. Services like `sslip.io` and `traefik.me` provide wildcard DNS that resolves based on the IP address in the hostname: - `sslip.io`: If your ingress IP is `35.X.X.X`, then `myapp.35.X.X.X.sslip.io` resolves to `35.X.X.X` - `traefik.me`: Similar functionality but with dashes, e.g., `myapp.35-X-X-X.traefik.me` These services are useful for: - Local development with Minikube - Quick testing in cloud environments before setting up proper DNS - Temporary deployments or demos The example values files below use `sslip.io` for demonstration purposes. For production, you should use proper DNS records in your cloud provider. **Environment Configuration** In the next step you will need to decide on which environment to deploy to and configure your values files accordingly before installation. :::info Available configuration options The examples below provide minimal configurations to get you started quickly. For a complete list of all available configuration options and advanced customization, refer to the [full values.yaml file](https://github.com/growthbook/growthbook/tree/main/charts/growthbook/values.yaml) in our repository. ::: ## Environment-Specific Values Default hostnames below use sslip.io for quick testing. For production, use a domain registered in your cloud provider's DNS (e.g., Route53 for AWS). The hostnames in the values file must match the domain you intend to use for Ingress. Replace INGRESS_IP with the external IP assigned to your Ingress object. Minimal values-aws.yaml ```yaml global: env: - name: APP_ORIGIN # For testing only! Replace with your real domain for production. value: http://growthbook..sslip.io frontend: service: name: frontend port: 3000 targetPort: 3000 env: - name: API_HOST # For testing only! Replace with your real domain for production. value: http://api..sslip.io backend: mongodbEnabled: true service: name: backend port: 3100 targetPort: 3100 env: - name: JWT_SECRET valueFrom: secretKeyRef: name: jwt-secret key: jwt-secret - name: ENCRYPTION_KEY valueFrom: secretKeyRef: name: encryption-key key: encryption-key - name: UPLOAD_METHOD value: s3 # ===== S3 uploads configuration ===== - name: S3_BUCKET value: "BUCKET_NAME" # default: us-east-1 - name: S3_REGION value: "S3_REGION" # default: https://${S3_BUCKET}.s3.amazonaws.com/ - name: S3_DOMAIN value: "https://${S3_BUCKET}.s3.${S3_REGION}.amazonaws.com/" mongodb: enabled: true ingress: enabled: true className: "alb" annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/scheme: internet-facing hosts: - host: growthbook..sslip.io # For testing only! Replace for production. paths: - path: /* pathType: ImplementationSpecific service: frontend - host: api..sslip.io # For testing only! Replace for production. paths: - path: /* pathType: ImplementationSpecific service: backend tls: [] ``` Installation: ```bash helm install growthbook oci://ghcr.io/growthbook/charts/growthbook -f values-aws.yaml ``` After Install: Update Hostnames 1. Wait for your Ingress object to be assigned an external IP. This might take anywhere from 3 to 5 minutes. 2. Update your values-aws.yaml file to use this IP in all sslip.io hostnames. 3. Run helm upgrade --install growthbook oci://ghcr.io/growthbook/charts/growthbook -f values-aws.yaml to apply the change. Default hostnames below use sslip.io for quick testing. For production, use a domain registered in your cloud provider's DNS (e.g., Cloud DNS for GCP). The hostnames in the values file must match the domain you intend to use for Ingress. Replace INGRESS_IP with the external IP assigned to your Ingress object. Minimal values-gcp.yaml ```yaml global: env: - name: APP_ORIGIN # For testing only! Replace with your real domain for production. value: http://growthbook..sslip.io frontend: service: annotations: cloud.google.com/neg: '{"ingress": true}' name: frontend port: 3000 targetPort: 3000 env: - name: API_HOST # For testing only! Replace with your real domain for production. value: http://api..sslip.io backend: mongodbEnabled: true volumeClaim: enabled: false service: annotations: cloud.google.com/neg: '{"ingress": true}' name: backend port: 3100 targetPort: 3100 env: - name: JWT_SECRET valueFrom: secretKeyRef: name: jwt-secret key: jwt-secret - name: ENCRYPTION_KEY valueFrom: secretKeyRef: name: encryption-key key: encryption-key - name: UPLOAD_METHOD value: google-cloud - name: GCS_BUCKET_NAME value: "BUCKET_NAME" - name: GCS_DOMAIN value: "https://storage.googleapis.com/BUCKET_NAME/" mongodb: enabled: true ingress: enabled: true className: "gce" annotations: kubernetes.io/ingress.class: gce hosts: - host: growthbook..sslip.io # For testing only! Replace for production. paths: - path: /* pathType: ImplementationSpecific backend: service: name: frontend port: number: 3000 - host: api..sslip.io # For testing only! Replace for production. paths: - path: /* pathType: ImplementationSpecific backend: service: name: backend port: number: 3100 tls: [] ``` Installation: ```bash helm install growthbook oci://ghcr.io/growthbook/charts/growthbook -f values-gcp.yaml ``` After Install: Update Hostnames 1. Wait for your Ingress object to be assigned an external IP. 2. Update your values-gcp.yaml file to use this IP in all sslip.io hostnames. 3. Run helm upgrade --install growthbook oci://ghcr.io/growthbook/charts/growthbook -f values-gcp.yaml to apply the change. Minikube is for local testing and development only, not for production. Default hostnames below use sslip.io for quick testing. Replace MINIKUBE-IP with the output of minikube ip. Tip: Local TLS with mkcert For a more realistic development setup, you can use mkcert to issue locally-trusted TLS certificates within Minikube. This allows you to work with HTTPS without browser warnings. Click for mkcert setup instructions **Prerequisites** Before starting the setup, make sure you have Minikube installed and running. You'll also need `kubectl` and `helm` available on your system. **Get Your Minikube IP** Start Minikube if it's not already running: ```bash minikube start ``` Get your Minikube IP address: ```bash minikube ip ``` Make note of this IP address - you'll need to replace all instances of `${MINIKUBE_IP}` in the values file with this actual IP. **Enable NGINX Ingress Controller** The setup requires NGINX Ingress Controller. Enable it in Minikube: ```bash minikube addons enable ingress ``` **Install cert-manager** Install cert-manager to manage TLS certificates in your cluster: ```bash # Create the namespace kubectl create namespace cert-manager # Install cert-manager using the official manifest kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.0/cert-manager.yaml # Wait for cert-manager pods to be ready kubectl get pods -n cert-manager --watch ``` **Install mkcert** Install `mkcert` on your local machine. On macOS, you can use Homebrew: ```bash brew install mkcert ``` For other operating systems, follow the [mkcert installation instructions](https://github.com/FiloSottile/mkcert#installation). Install the local CA in your system trust store: ```bash mkcert -install ``` **Create CA Secret and ClusterIssuer** Create a Kubernetes secret containing your local CA: ```bash kubectl create secret tls mkcert-ca-key-pair \ --key "$(mkcert -CAROOT)"/rootCA-key.pem \ --cert "$(mkcert -CAROOT)"/rootCA.pem \ -n cert-manager ``` Create a ClusterIssuer that will use this CA to issue certificates: ```bash kubectl apply -f - Minimal values-minikube.yaml (with TLS) ```yaml global: security: allowInsecureImages: true env: # Replace ${MINIKUBE-IP} with the output of `minikube ip` - name: APP_ORIGIN value: https://growthbook.${MINIKUBE-IP}.sslip.io frontend: service: name: frontend port: 3000 targetPort: 3000 env: # Replace ${MINIKUBE-IP} with the output of `minikube ip` - name: API_HOST value: https://api.${MINIKUBE-IP}.sslip.io backend: mongodbEnabled: true volumeClaim: enabled: true service: name: backend port: 3100 targetPort: 3100 env: - name: JWT_SECRET valueFrom: secretKeyRef: name: jwt-secret key: jwt-secret - name: ENCRYPTION_KEY valueFrom: secretKeyRef: name: encryption-key key: encryption-key - name: UPLOAD_METHOD value: local mongodb: enabled: true persistence: enabled: true # Only needed if running Minikube on Apple Silicon (M1/M2/M3) # image: # registry: docker.io # repository: dlavrenuek/bitnami-mongodb-arm # tag: 7.0.15 ingress: className: "nginx" annotations: # This tells cert-manager to use our mkcert issuer cert-manager.io/cluster-issuer: mkcert enabled: true hosts: - host: growthbook.${MINIKUBE-IP}.sslip.io paths: - path: / pathType: ImplementationSpecific service: frontend - host: api.${MINIKUBE-IP}.sslip.io paths: - path: / pathType: ImplementationSpecific service: backend tls: - secretName: growthbook-tls hosts: - growthbook.${MINIKUBE-IP}.sslip.io - api.${MINIKUBE-IP}.sslip.io ``` Installation: ```bash helm install growthbook oci://ghcr.io/growthbook/charts/growthbook -f values-minikube.yaml ``` ## Accessing GrowthBook After deploying with Helm, access GrowthBook using the Ingress hostnames you configured in your values file: - **Minikube:** - Run `minikube tunnel` in a separate terminal to expose the Ingress IP to your host. - Frontend: `https://growthbook.${MINIKUBE-IP}.sslip.io` - Backend/API: `https://api.${MINIKUBE-IP}.sslip.io` - (Replace `${MINIKUBE-IP}` with the output of `minikube ip`) When running Minikube on MacOS with Docker Desktop, you may need to add some rules to your `/etc/hosts` file: ```bash echo "127.0.0.1 growthbook.${MINIKUBE-IP}.sslip.io" | sudo tee -a /etc/hosts echo "127.0.0.1 api.${MINIKUBE-IP}.sslip.io" | sudo tee -a /etc/hosts ``` - **AWS & GCP** - Use the hostname you set in your values file - Check your Ingress/ALB resource for the external DNS name if using AWS Load Balancer Controller or domain if using a cloud-managed Ingress > Open the frontend URL in your browser to start using GrowthBook. The backend/API URL is used by the frontend and for API access. ## Troubleshooting If something isn't working as expected, use these commands to check the status of your resources: - List all pods and their status: ```bash kubectl get pods -A ``` - Describe a specific pod (replace `POD_NAME` and `NAMESPACE`): ```bash kubectl describe pod POD_NAME -n NAMESPACE ``` - Check the status of your services: ```bash kubectl get svc -A ``` - Inspect your Ingress resources: ```bash kubectl get ingress -A kubectl describe ingress INGRESS_NAME -n NAMESPACE ``` --- ## Production # Production Best Practices When properly configured, a self-hosted GrowthBook deployment can scale to billions of requests per month. The default settings in the `docker-compose` file are meant to get you up and running quickly on a local dev machine. There are a few things to keep in mind when deploying GrowthBook securely at scale in production. ## Security There are a number of best practices to keep your GrowthBook deployment secure. ### Encryption Keys and Secrets First, make sure you pick long, random strings for your encryption keys and secrets. Specifically, there are 3 environment variables that need to be configured: - **NODE_ENV** - Set to `production` to turn on additional security checks and logging - **JWT_SECRET** - Auth signing key (use a long random string) - **ENCRYPTION_KEY** - Data source credential encryption key (use a long random string) If you change the `ENCRYPTION_KEY`, you will need to migrate any existing data sources using the following script: ```bash # If you didn't have an ENCRYPTION_KEY before, leave OLD_KEY blank below docker compose run --workdir /usr/local/src/app/packages/back-end \ growthbook node dist/scripts/migrate-encryption-key.js OLD_KEY ``` ### Data Sources In general, GrowthBook only requires read access to your data warehouse. The one exception is if Pipeline Mode is enabled, in which case GrowthBook will need write access to a specific schema that you specify in order to create temporary tables for experiment analysis. We do our best to protect against SQL injection attacks and destructive queries, but because a large part of GrowthBook's functionality is running arbitrary user-defined SQL, there is always a risk. For this reason, we recommend following the principle of least privilege and creating a dedicated database user for GrowthBook with only the necessary permissions. That means only read access to your data, plus write access to a dedicated schema created specifically for GrowthBook (if Pipeline Mode is enabled). ### Networking The most secure thing you can do is to put your entire GrowthBook deployment behind a firewall and/or corporate VPN. This way, you can restrict access to only those who need to sign into the GrowthBook platform. This can drastically reduce the attack vector and serves as an important first layer of security. If you are using GrowthBook to serve feature flags to client-side apps, you will need some way for your end users to download the feature flag payload outside of the firewall or VPN. There are a few secure approaches for this: 1. Use a [**GrowthBook Proxy**](/self-host/proxy) server. This way, your main GrowthBook instance can remain on a private subnet and only the Proxy needs to be exposed to the internet. The Proxy has a very small attack surface compared to the full GrowthBook app. 2. Use a [**CDN**](/self-host/cdn). If you go this route, it's very important to only allow requests to the `/api/features/*` endpoint in your CDN. If you allow access to other endpoints (e.g. `/auth/*` or other API routes) you lose many of the benefits of a Firewall or VPN and may even end up leaking sensitive information. 3. Use **SDK Webhooks** to send your feature payload to a publicly accessible place (e.g. a public S3 bucket). This switches GrowthBook from a "pull" model to a "push" model and you no longer need to allow any incoming requests from users at all. ### Authentication and Permissions For our Enterprise customers, we recommend using [SSO](/sso) and [SCIM](/integrations/scim) to authenticate with GrowthBook and manage permissions. This offloads access control and management to your existing identity management infrastructure and further reduces the possible attack surface. No matter which authentication method you use, it's best to follow the **least access** principle. Admins can invite other users and grant permissions, so restrict this role to only the people who really need it. Read more about [permissions](/account/user-permissions). When using our REST API, it's best to use **readonly** API keys when possible. When you need write access, using **Personal Access Tokens** is best since it inherits the permissions of the user who created it. Only use admin Secret Access Tokens as a last resort. ### MongoDB Securing MongoDB in production is its own topic entirely and outside the scope of GrowthBook. We recommend using a hosted solution such as [MongoDB Atlas](https://www.mongodb.com/atlas/database), which is secure by default. If you do want to host MongoDB yourself in production, they have a [detailed guide](https://www.mongodb.com/docs/manual/administration/production-notes/) on their docs. ## Email SMTP Settings Configuring email is required in order to send experiment alerts, team member invites, and reset password emails. - **EMAIL_ENABLED** ("true" or "false") - **EMAIL_HOST** - **EMAIL_PORT** - **EMAIL_HOST_USER** - **EMAIL_HOST_PASSWORD** - **EMAIL_FROM** ## File Uploads GrowthBook lets users add Markdown comments throughout the app and these support image uploads. The **UPLOAD_METHOD** environment variable controls where to store uploaded files and screenshots. The supported values are `local`, `s3`, and `google-cloud`. ### local This is the default value. Uploads are stored in the GrowthBook docker container at `/usr/local/src/app/packages/back-end/uploads`. In production, you should mount a shared volume here to persist uploads across container restarts and ensure your horizontally scaled GrowthBook instances all share the same uploads directory. ### s3 Store uploads in an AWS S3 bucket. - **S3_BUCKET** - **S3_REGION** (defaults to `us-east-1`) - **S3_DOMAIN** (defaults to `https://${S3_BUCKET}.s3.amazonaws.com/`) - **AWS_ACCESS_KEY_ID** (not required when deployed to AWS with an instance role) - **AWS_SECRET_ACCESS_KEY** (not required when deployed to AWS with an instance role) ### google-cloud Store uploads in a Google Cloud Storage bucket. - **GCS_BUCKET_NAME** - **GCS_DOMAIN** (defaults to `https://storage.googleapis.com/${GCS_BUCKET_NAME}/`) - **GOOGLE_APPLICATION_CREDENTIALS** (not required when deployed to GCP with an instance service account) ## Scaling ### GrowthBook Container GrowthBook container instances are stateless and can be horizontally scaled behind a load balancer. We recommend each instance have at least 2GB of RAM and 1vCPU (some may need more RAM not to get Out Of Memory errors). You should run at least 3 instances at a time for maximum fault tolerance. The most intensive part of GrowthBook is the stats engine and background jobs. If you are running a high volume of experiments, processing those experiments may impact other requests to growthbook. If you notice this, you can separate concerns by starting one "Jobs Server" with increased resourced as needed and the env vars `PYTHON_SERVER_MODE=true`, `GB_STATS_ENGINE_POOL_SIZE=` on the "Jobs Server" and the others with `GB_STATS_ENGINE_MIN_POOL_SIZE=0`, `EXTERNAL_PYTHON_SERVER_URL=`, and `CRON_DISABLED=true`. The "Jobs Server" `/stats` endpoint has no built-in authentication, so it should only be reachable on a private network. As defense-in-depth, set a shared secret with `PYTHON_SERVER_AUTH_TOKEN=` on both the "Jobs Server" and the main instances — when set, the "Jobs Server" rejects any request to `/stats` that doesn't present it as a `Bearer` token. ### MongoDB All application state is stored in MongoDB. A small GrowthBook deployment can run with shared CPU/RAM and under 512MB of storage. A very large deployment with 1,000 feature flags and experiments will take up around 5GB of storage and require at least 2GB of RAM and 2vCPUs. The easiest option in production is to use a managed service: - [MongoDB Atlas](https://www.mongodb.com/) (recommended) - AWS DocumentDB - [FerretDB](/self-host/ferretdb) - Azure CosmosDB If you prefer to run MongoDB yourself, we recommend running a replica set configuration with 3 nodes. This will ensure that your data is always available even if one of your MongoDB instances goes down. ### Serving Feature Flags Hitting the GrowthBook instance directly from our SDKs can work fine at a small scale, but we recommend an additional layer for added scalability and fault tolerance. There are a few options for this: 1. Use a [**GrowthBook Proxy**](/self-host/proxy) server. Each Proxy Server instance can easily handle thousands of requests per second. We recommend horizontally scaling these instances and configuring Redis to keep them in sync. 2. Use a **CDN** in front of the `/api/features/*` endpoints. The response from GrowthBook contains cache headers by default that most CDNs will recognize without any custom configuration. You can tweak these cache settings with the following environment variables (all in seconds): - `CACHE_CONTROL_MAX_AGE` (default `30`) - `CACHE_CONTROL_STALE_WHILE_REVALIDATE` (default `3600` = 1 hour) - `CACHE_CONTROL_STALE_IF_ERROR` (default `36000` = 10 hours) 3. Use **SDK Webhooks** to send your feature payload to a publicly accessible place (e.g. an S3 bucket) and point the SDKs there instead. This bypasses the issue by switching GrowthBook from a "pull" model to a "push" model. ## Observability and Logging The GrowthBook API is instrumented with OpenTelemetry to publish observability metrics, traces, and logs. To enable, you must set the environment variable `TRACING_PROVIDER=opentelemetry` The standard [OTEL\_\* Environment Variables](https://opentelemetry.io/docs/concepts/sdk-configuration/) are supported, such as `OTEL_SERVICE_NAME` and `OTEL_EXPORTER_OTLP_ENDPOINT`. In addition to this, make sure to configure the **EXPRESS_TRUST_PROXY_OPTS** environment variable. It supports boolean (true/false), string values, and integer values for trusting the _n_th hop from the front-facing proxy server as the client. Leavy empty or specify `false` to use Express' default behavior. If you are running GrowthBook behind a proxy or load balancer, this is required to track the correct user IP for audit log events. Read more about this setting in [Express' documentation](https://expressjs.com/en/5x/api.html#trust.proxy.options.table). --- ## GrowthBook Proxy # GrowthBook Proxy The GrowthBook Proxy server sits between your application and GrowthBook (both Cloud or Self-hosted instances). It turbocharges your GrowthBook implementation by providing **speed**, **scalability**, **security**, and **real-time** feature rollouts. The GrowthBook Proxy does require spinning up infrastructure and all of the maintenance/devops costs associated with that. If you are looking for a simpler hands-off alternative, you can use a [CDN](/self-host/cdn) instead. ## Features - **Caching** - Significantly faster feature lookups! - In-memory cache plus an optional distributed layer (Redis or MongoDB) - Automatic cache invalidation when features change in GrowthBook (using WebHooks) - **Streaming** - Updates your application in real-time as features are changed or toggled in GrowthBook (Javascript and React only) - **Remote Evaluation** - Hide your features' business logic in insecure environments - **Security** - Private-key authentication between GrowthBook and GrowthBook Proxy - **Scalability** - Support millions of concurrent users ## Installation ### Using docker compose If you are already using `docker compose` to run GrowthBook, we have a pre-configured setup that includes a GrowthBook Proxy instance. Just run ```bash docker compose -f docker-compose.proxy.yml up -d ``` This will start the proxy server on port 3300. Check `http://localhost:3300/healthcheck` to ensure it's working correctly. ### Standalone You can also run the GrowthBook Proxy as a standalone Docker container. First, pull the latest image ```bash docker pull growthbook/proxy:latest ``` Then, run a GrowthBook Proxy instance on port 3300 ```bash docker run -d -p 3300:3300 \ -e "GROWTHBOOK_API_HOST=https://growthbook-api.example.com" \ -e "SECRET_API_KEY=something_secret" \ --name gbproxy growthbook/proxy ``` Check `http://localhost:3300/healthcheck` to ensure it's running correctly. ### Authentication You will need to create a "readonly" secret API key in GrowthBook by going to **Settings → API Keys** (you can also use a Personal Access Token if preferred). Or you can use a custom `SECRET_API_KEY` of your choosing. Whichever method you choose, this key will be used to authenticate your proxy server with the GrowthBook app. Lastly, for self-hosted customers, add environment variables your main GrowthBook API server to enable the proxy: ```bash PROXY_ENABLED=1 PROXY_HOST_PUBLIC=https://growthbook-proxy.example.com # OPTIONAL: You can either create the secret key in the GrowthBook UI or define one here or SECRET_API_KEY=something_secret ``` Note: Setting `PROXY_HOST_PUBLIC` is not strictly required, but is considered a best practice. Setting it enables faster rollouts by allowing GrowthBook to push updates to your proxy whenever feature definitions change. ### Cloud Customers For cloud customers who self-host a proxy server, you must configure each SDK Connection to use the proxy server. You may optionally enter your proxy server's public host URL. This enables faster rollouts by allowing GrowthBook to push updates to your proxy whenever your feature definitions change. If you do not provide a proxy host URL, your proxy server the proxy will fall back to a pull-based stale-while-revalidate caching strategy. Go to **SDK Configuration → SDK Connections** in the GrowthBook app to configure the proxy server per each connection. ## Using with the SDKs The GrowthBook Proxy has the same public feature endpoints as GrowthBook, so all you need to do is change the API host your SDK clients connect to: ```ts // Before const gb = new GrowthBook({ apiHost: "https://growthbook-api.example.com", clientKey: "sdk-abc123" }); // After (clientKey remains the same) const gb = new GrowthBook({ apiHost: "https://growthbook-proxy.example.com", clientKey: "sdk-abc123" }); ``` ## Configuration The GrowthBook Proxy supports a number of configuration options available via environment variables. Some of the more common options are: - `GROWTHBOOK_API_HOST` - Set this to the host and port of your GrowthBook API instance - `SECRET_API_KEY` - Create a secret API key in GrowthBook by going to **Settings → API Keys** - `NODE_ENV` - Set to "production" to hide debug and informational log messages - `CACHE_ENGINE` - One of - `memory`, `redis`, or `mongo` - `CACHE_CONNECTION_URL` - The URL of your redis or mongo cluster (if using) - `CACHE_STALE_TTL` - Number of seconds until a cache entry is considered stale - `CACHE_EXPIRES_TTL` - Number of seconds until a cache entry is expired You can also configure the GrowthBook Proxy to handle SSL termination. It supports HTTP/2 by default, which is required for high performance streaming. - `USE_HTTP2` - Set to "true" or "1" to enable - `HTTPS_CERT` - The SSL certificate - `HTTPS_KEY` - The SSL key For more complete configuration documentation, please see the GrowthBook Proxy GitHub page: https://github.com/growthbook/growthbook-proxy ## Best Practices In high-traffic production scenarios, there are a few best practices to follow - Auto-scale GrowthBook Proxy instances based on number of active connections or memory - Run the instances in the same region as your application servers for the lowest latency - Add a load balancer in-front that supports HTTP/2 and streaming responses (AWS ALB, HAProxy, etc.) - Use Redis (or MongoDB) as the cache engine for more consistent feature releases - Use the `/healthcheck` endpoint to determine if the instances are running correctly --- ## Remote Evaluation # Remote Evaluation Remote Evaluation brings the security benefits of a backend SDK to client-side environments by evaluating feature flags exclusively on a private server. This ensures that sensitive information within targeting rules and unused features and experiment variations are never exposed to the client. ## Overview When using Remote Evaluation, your client-side SDK sends user attributes to a remote evaluation endpoint, which then returns the evaluated feature flag values for that specific user. The primary benefit is **security**: sensitive targeting rules, unused variations, business logic, and experiment configuration details remain hidden on your server and never get exposed to the client. Clients only receive the evaluated feature flag values they need for their specific attributes. ### Trade-offs Remote Evaluation improves security through feature obfuscation, but this comes with trade-offs: - **Added Latency** - Each evaluation requires a network request to your remote evaluation endpoint, adding latency compared to local evaluation. - **Reduced Cacheability** - The SDK payload cannot be cached and reused across users or when user attributes change; each new user or new set of attributes requires a new request to the remote evaluation endpoint. - **Increased Infrastructure** - Requires running and maintaining a GrowthBook Proxy Server, edge worker, or private evaluation endpoint. Remote Evaluation is ideal when SDK payload security is paramount and the latency trade-off is acceptable for your use case. :::note When to Use Remote Evaluation Remote Evaluation is designed for **client-side environments** (web browsers, mobile apps, desktop apps) where code and data are exposed to end users. It should **not** be used in backend contexts where your SDK already runs in a secure server environment. ::: ## How It Works Remote Evaluation changes what data flows between your client, evaluation endpoint, and GrowthBook: **Traditional (Local Evaluation):** 1. Client SDK fetches the entire feature payload from GrowthBook (contains all feature rules, targeting conditions, experiment configurations, and variations) 2. Client evaluates features locally using user attributes 3. All targeting logic and unused variations are visible in the client **Remote Evaluation:** 1. Client SDK sends user attributes to your remote evaluation endpoint. 2. Endpoint periodically fetches and caches the feature payload from GrowthBook (non-blocking). 3. Endpoint evaluates features server-side using the provided user attributes. 4. Endpoint returns evaluated feature values and scrubbed experiment metadata to the client. Targeting rules, unused variations, and experiment configurations never reach the client. 5. Client SDK fires any experiment tracking callbacks (experiment exposure events); these are deferred by the remote evaluation endpoint and hydrated back to the client and are only fired when needed. This architecture ensures that sensitive business logic remains on your server while clients receive only the evaluated results they need. Experiment tracking callbacks execute on the client, allowing you to send exposure events to your existing analytics tools. ## Supported SDKs The following SDKs support Remote Evaluation (see each SDK's documentation for setup instructions): - [**JavaScript**](/lib/js#remote-evaluation) (v0.29.0+) - [**React**](/lib/react#remote-evaluation) (v0.19.0+) - [**HTML Script Tag**](/lib/script-tag) - [**Android**](/lib/kotlin#remote-evaluation) (v1.1.50+) - [**iOS**](/lib/swift#remote-evaluation-vs-local-evaluation) (v1.0.50+) - [**Java**](/lib/java) (v0.9.92+) - [**Flutter**](/lib/flutter#remote-evaluation) (v3.7.0+) - [**C#**](/lib/csharp#remote-evaluation) (v1.1.3) :::tip Check the SDK version requirements in your specific SDK's documentation. Remote Evaluation must be explicitly enabled in your SDK Connection settings in GrowthBook. ::: ## Installation Options Several options are available for self-hosting a Remote Evaluation endpoint or service. Choose the GrowthBook Proxy for a full-featured solution with caching and streaming, edge workers for global low-latency, or build a custom endpoint to integrate with your existing infrastructure. ### 1. GrowthBook Proxy {#growthbook-proxy} The **GrowthBook Proxy** is a standalone server that sits between your application and GrowthBook, providing a caching layer which supports both streaming and remote evaluation. Remote evaluation is enabled by default on the GrowthBook Proxy. If you want to use sticky bucketing with remote evaluation, you must configure a Redis store for your proxy to use for user sticky bucket storage. See the [GrowthBook Proxy documentation](/self-host/proxy) for complete details. ### 2. Edge Worker {#edge-worker} You may configure an **edge worker** to host a remote evaluation endpoint distributed on an edge network, providing a lightweight solution with lower network latency. An edge remote evaluation implementation requires implementing the [`@growthbook/proxy-eval`](https://www.npmjs.com/package/@growthbook/proxy-eval) NPM package ([source](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/eval)) within an edge runtime. This library is supported across all major edge providers. A [Cloudflare example project](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/eval/example/cloudflare) is available to help you get started. :::note Sticky Bucketing Limitations Sticky bucketing for remote evaluation on edge currently has limited support. Contact us for more information if you need this functionality. ::: ### 3. Custom Remote Evaluation Endpoint {#remote-evaluation-endpoint} You may build your own remote evaluation endpoint using the [`@growthbook/proxy-eval`](https://www.npmjs.com/package/@growthbook/proxy-eval) NPM package ([source](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/eval)). This library provides the core remote evaluation logic and can be integrated into any Node.js environment, including: - Express or other Node.js web servers - Serverless functions (AWS Lambda, Google Cloud Functions, etc.) - Other backend JavaScript runtimes This approach gives you complete control over your remote evaluation infrastructure, allowing you to integrate with existing authentication, logging, and monitoring systems. **Example Express integration:** ```typescript import express from "express"; import cors from "cors"; import { evaluateFeatures } from "@growthbook/proxy-eval"; const app = express(); app.use(cors()); app.use(express.json()); // Cache the GrowthBook payload let cachedPayload: any = null; let lastFetch = 0; const CACHE_TTL = 60 * 1000; // 1 minute async function getPayload() { // Refresh cache if stale if (!cachedPayload || Date.now() - lastFetch > CACHE_TTL) { const response = await fetch( "https://cdn.growthbook.io/api/features/sdk-abc123" ); cachedPayload = await response.json(); lastFetch = Date.now(); } return cachedPayload; } app.post("/api/eval", async (req, res) => { try { const { attributes = {}, forcedVariations = {}, forcedFeatures = [], url = "", } = req.body; const forcedFeaturesMap = new Map(forcedFeatures); const payload = await getPayload(); const evalResponse = await evaluateFeatures({ payload, attributes, forcedVariations, forcedFeatures: forcedFeaturesMap, url, }); res.json(evalResponse); } catch (error) { console.error("Evaluation error:", error); res.status(500).json({ error: "Evaluation failed" }); } }); app.listen(3000, () => { console.log("Remote evaluation endpoint running on port 3000"); }); ``` For a complete Cloudflare Worker example, see the [reference implementation](https://github.com/growthbook/growthbook-proxy/blob/main/packages/lib/eval/example/cloudflare/src/index.ts). See the [`@growthbook/proxy-eval` documentation](https://github.com/growthbook/growthbook-proxy/tree/main/packages/lib/eval) for complete integration details and configuration options. #### Debug Endpoint for Self-Hosted GrowthBook If you're running a self-hosted GrowthBook instance, a debug remote evaluation endpoint is available at: ``` POST https://your-growthbook-api.example.com/api/eval/:clientKey ``` This endpoint is provided for debugging and testing your remote evaluation connection. It is **not designed for production scale** and should only be used during development. For production workloads, use the GrowthBook Proxy, an edge worker, or build a custom endpoint as described above. --- ## SSO Instructions import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; # Enterprise SSO SSO is available on GrowthBook Cloud or Self-hosted via OpenID Connect. If you are using the Cloud, your account representative will help you get this setup, though the steps are mostly the same. To enable SSO on your self hosted GrowthBook instance, you will need an active license key and then you may add the SSO settings for your provider. If your provider is not listed below, you can use the generic Open ID Connect. For GrowthBook Cloud, you will need to send your account representative the following: **CLIENT_ID**, **CLIENT_SECRET**, **EMAIL_DOMAIN**, what provider you're using, and for some providers, the **TENANT_ID**. You can use the instructions below to get these values. If you are self-hosting, you can use the instructions below to create a JSON object with the settings, then it should be JSON encoded and then set to the environment variable `SSO_CONFIG`. ## Generic Open ID Connect - **LICENSE_KEY** - Your signed license key provided by the GrowthBook team - **SSO_CONFIG** - A JSON-encoded string that configures SSO (using OpenID Connect). It should be an object with the following keys: - `clientId` (string) - `clientSecret` (string) - `emailDomains` (array of strings, optional) - Allow [auto-joining](/account/user-permissions#self-registering-and-automatic-approvals) from a specified email domain. - `metadata` (object) - `issuer` (string) - `authorization_endpoint` (string) - `jwks_uri` (string) - `id_token_signing_alg_values_supported` (array of strings) - `token_endpoint` (string) - `code_challenge_methods_supported` (array of strings) - `logout_endpoint` (string, optional) - `extraQueryParams` (object, optional) - Dictionary of extra query params to be passed along with the `/authorize` OAuth call - `additionalScope` (string, optional) - Additional scopes to include, along with the default value: `openid profile email` For SSO, make sure the following callback URL is whitelisted: - `{APP_ORIGIN}/oauth/callback` For the best SSO user experience, enable offline access and refresh tokens in your Identity Provider. :::info Configuring SSO With all the SSO providers listed below, replace the all caps values with values from the provider, JSON encode the object, and set to the `SSO_CONFIG` environment variable for GrowthBook. ::: ## SSO Providers ### Okta ```json { "clientId": "CLIENT_ID", "clientSecret": "CLIENT_SECRET", "emailDomains": ["EMAIL_DOMAIN"], "additionalScope": "offline_access", "metadata": { "issuer": "BASE_URL", "authorization_endpoint": "BASE_URL/oauth2/v1/authorize", "id_token_signing_alg_values_supported": [ "RS256" ], "jwks_uri": "BASE_URL/oauth2/v1/keys", "token_endpoint": "BASE_URL/oauth2/v1/token", "code_challenge_methods_supported": [ "S256" ] } } ``` See complete Okta instructions Create an OIDC Web application: ![Create an OIDC Web application](/images/guides/SSO-Okta-1.png) Allow refresh tokens and specify callback URLs. If you are using the cloud, you can use the following values: Sign-in redirect URIs - https://app.growthbook.io/oauth/callback Sign-out redirect URIs (optional) - https://app.growthbook.io If you are self-hosting, replace with https://app.growthbook.io with the value from your APP_ORIGIN ![Okta settings](/images/guides/SSO-Okta-2.png) Require PKCE as additional verification(optional) ![Require PKCE](/images/guides/SSO-Okta-3.png) You will need the following in order to configure SSO in GrowthBook: CLIENT_ID CLIENT_SECRET BASE_URL - This is typically in the format: https://your-company.okta.com/ or https://your-company.oktapreview.com/ EMAIL_DOMAIN If using GrowthBook Cloud, send your account representative the above and we will enable SSO on your account. If self-hosting, add the environment settings at the beginning of this section to enable SSO on your instance. ### Google ```json { "clientId": "CLIENT_ID", "clientSecret": "CLIENT_SECRET", "emailDomains": ["EMAIL_DOMAIN"], "metadata": { "issuer": "https://accounts.google.com", "authorization_endpoint": "https://accounts.google.com/o/oauth2/v2/auth", "token_endpoint": "https://oauth2.googleapis.com/token", "jwks_uri": "https://www.googleapis.com/oauth2/v3/certs", "id_token_signing_alg_values_supported": [ "RS256" ], "code_challenge_methods_supported": [ "S256" ] }, "extraQueryParams": { "access_type": "offline", "prompt": "consent" } } ``` ### Auth0 ```json { "clientId": "CLIENT_ID", "clientSecret": "CLIENT_SECRET", "emailDomains": ["EMAIL_DOMAIN"], "additionalScope": "offline_access", "metadata": { "issuer": "https://TENANT.auth0.com/", "authorization_endpoint": "https://TENANT.auth0.com/authorize", "logout_endpoint": "https://TENANT.auth0.com/v2/logout?client_id=CLIENT_ID", "id_token_signing_alg_values_supported": [ "HS256", "RS256" ], "jwks_uri": "https://TENANT.auth0.com/.well-known/jwks.json", "token_endpoint": "https://TENANT.auth0.com/oauth/token", "code_challenge_methods_supported": [ "S256", "plain" ], "audience": "AUDIENCE" } } ``` :::note Enabling offline access and OIDC compliance When setting up Auth0, please ensure that you've enabled offline access, and check the `OIDC Compliant` checkbox. ::: ### Azure AD ```json { "clientId": "CLIENT_ID", "clientSecret": "CLIENT_SECRET", "emailDomains": ["EMAIL_DOMAIN"], "additionalScope": "offline_access", "metadata": { "token_endpoint": "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token", "jwks_uri": "https://login.microsoftonline.com/TENANT_ID/discovery/v2.0/keys", "id_token_signing_alg_values_supported": ["RS256"], "code_challenge_methods_supported": ["S256"], "issuer": "https://login.microsoftonline.com/TENANT_ID/v2.0", "authorization_endpoint": "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/authorize", "logout_endpoint": "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/logout" } } ``` :::note Registering an Application in Azure In Azure, register an Application, instead of Enterprise, as we use OpenID Connect, not SAML. ::: See complete Azure instructions Register an Application (we use OpenID Connect, so choose regular app, not Enterprise) ![Register an Application](/images/guides/SSO-Azure-1.png) Enter the redirect URL as APP_HOST/oauth/callback ![Redirect URL](/images/guides/SSO-Azure-2.png) Take note of your Application Id (CLIENT_ID) and Directory Id (TENANT_ID). You will need it later ![Application Id](/images/guides/SSO-Azure-3.png) Generate a new Client Secret ![Client Secret](/images/guides/SSO-Azure-4.png) Take note of the Secret Value (CLIENT_SECRET). You will need it in the next step Construct the JSON configuration for GrowthBook. Replace CLIENT_ID, CLIENT_SECRET, EMAIL_DOMAIN, and TENANT_ID, as into the JSON object above. Pass the JSON string into the environment variable SSO_CONFIG of your GrowthBook container ### OneLogin ```json { "clientId": "CLIENT_ID", "clientSecret": "CLIENT_SECRET", "emailDomains": [ "EMAIL_DOMAIN" ], "additionalScope": "", "metadata": { "issuer": "https://[ONELOGIN_DOMAIN]/oidc/2", "authorization_endpoint": "https://[ONELOGIN_DOMAIN]/oidc/2/auth", "token_endpoint": "https://[ONELOGIN_DOMAIN]/oidc/2/token", "id_token_signing_alg_values_supported": [ "RS256", "HS256", "PS256" ], "jwks_uri": "https://[ONELOGIN_DOMAIN]/oidc/2/certs", "code_challenge_methods_supported": [ "S256" ], "logout_endpoint": "https://[ONELOGIN_DOMAIN]/oidc/2/logout" } } ``` See complete OneLogin instructions Create a new OIDC Web application. Browse to the Applications section of OneLogin from the top nav, then choose "Add App" from the top right. ![Create a new OIDC Web application](/images/guides/SSO-Onelogin-1.png) When the list of applications opens, search for "openid connect". Select the option named "OpenID Connect (OIDC)". ![Select OpenID Connect](/images/guides/SSO-Onelogin-2.png) Add the name "GrowthBook" (or whatever you would like to name it), an optional description and click save. The first save may not look like anything has happened, but you should see more options on the left menu when successful. Click on the "Configuration" item in the left nav menu, and enter the following values for the three input fields (note: for self-hosting, replace app.growthbook.io with your own domain) Login URL: https://app.growthbook.io Redirect URIs: https://app.growthbook.io/oauth/callback Post Logout Redirect URIs: https://app.growthbook.io ![OneLogin settings](/images/guides/SSO-Onelogin-3.png) You can optionally add images for the application from the info if you like. Click Save. Click on the "SSO" item from the left nav menu. Here you need to record the Client ID and Client Secret and Issuer URL ![OneLogin SSO settings](/images/guides/SSO-Onelogin-4.png) You can leave the other settings as default (Application Type: Web, and Authentication Method: Basic). Click save if you haven't already. For self-hosted instances: construct the JSON configuration for GrowthBook. Replace CLIENT_ID, CLIENT_SECRET, EMAIL_DOMAIN, and TENANT_ID, as into the JSON object above. The ONELOGIN_DOMAIN will be the same domain from your Issuer URL you recorded earlier. Pass the JSON string into the environment variable SSO_CONFIG of your GrowthBook container ### JumpCloud ```json { "clientId": "CLIENT_ID", "clientSecret": "CLIENT_SECRET", "emailDomains": [ "EMAIL_DOMAIN" ], "additionalScope": "offline_access", "metadata": { "token_endpoint": "https://oauth.id.jumpcloud.com/oauth2/token", "jwks_uri": "https://oauth.id.jumpcloud.com/.well-known/jwks.json", "id_token_signing_alg_values_supported": [ "RS256" ], "code_challenge_methods_supported": [ "S256" ], "issuer": "https://oauth.id.jumpcloud.com/", "authorization_endpoint": "https://oauth.id.jumpcloud.com/oauth2/auth", "logout_endpoint": "https://oauth.id.jumpcloud.com/oauth2/sessions/logout", "audience": "" } } ``` See complete JumpCloud instructions Create a new SSO Application. Browse to the User Authentication → SSO Applications from the left nav. Choose "Add new Application" from the top left. Choose a custom application by clicking 'select' under `custom application`, then click next then next again to confirm. ![JumpCloud custom application, page 1](/images/guides/SSO-jumpcloud-1.png) ![JumpCloud custom application, page 2](/images/guides/SSO-jumpcloud-2.png) You will then be asked what features you want to enable. Select `Manage Single Sign-On (SSO)`. Then, in the radio buttons, select "Configure SSO with OIDC". ![JumpCloud SSO configuration](/images/guides/SSO-jumpcloud-3.png) Add the name "GrowthBook" (or whatever you would like to name it), an optional description and click `next`. When you've confirmed the details, click `Configure Application`. ![JumpCloud SSO configuration with general info fields](/images/guides/SSO-jumpcloud-4.png) ![JumpCloud SSO configuration confirmation screen](/images/guides/SSO-jumpcloud-5.png) This will open a window letting you add additional configuration options for the GrowthBook application. Here are the settings you should set: Grant types: Select "Refresh Token" and Authentication Code Redirect URIs: https://app.growthbook.io/oauth/callback Client Authentication Type: Client Secret Basic Login URL: https://app.growthbook.io Attribute Mapping: Select both email and profile. The defaults for the fields that appear are all that is required. ![JumpCloud SSO OpenID Connect](/images/guides/SSO-jumpcloud-6.png) Once you have made the above settings, click 'Activate' from the bottom bar. You'll then be presented with a modal giving you the client id and client secret. Here you need to record the Client ID and Client Secret ![JumpCloud SSO client id and secret](/images/guides/SSO-jumpcloud-7.png) For self-hosted instances, construct the JSON configuration for GrowthBook. Replace CLIENT_ID, CLIENT_SECRET, and EMAIL_DOMAIN, as into the JSON object above. Pass the JSON string into the environment variable SSO_CONFIG of your GrowthBook container --- ## Aggregate Data # Aggregate Data and the Statistics Engine GrowthBook never collects user-level events or PII. To power our statistics engine, contained within the `gbstats` Python library, GrowthBook works solely with aggregate level data that is sufficient to conduct analyses of interest. ## Working with sums Specifically, GrowthBook mostly relies on queries that run on your data warehouse and return data aggregated across users. We aggregate data at the level of the experiment variation (or dimension-variation, if you have dimensions specified). We rely on sums, sums of squares, and sums of cross products to power our statistics engine. These simple quantities allow us to easily re-aggregate over different dimensions while still containing sufficient information to compute experiment statistics. Note that these are the aggregations at the experiment variation level---they are computed across users. These often follow an aggregation at the user level, which defaults to SUM for non-binomial metrics, or a 1 for binomial metrics. You may have overridden the user-level aggregation in your metric definition with some other function. However, when we then go to aggregate at the variation level, we sum across those user-level aggregations. ### Aggregate fields You can view the aggregate data we use for any metric in your experiment by selecting the three dots in the top right of the experiment table and clicking "View Queries". There, you can see the actual queries we build and the aggregate results they contain. The following image is an example of those aggregate values. ![View Query - SQL and aggregate statistics](/images/view-query-aggregate.png) The potential fields the queries will return are as follows, with all sums aggregating at the variation-dimension level: - `variation` - the name of the experiment variation - `dimension` - the name of the dimension we aggregate to; defaults to "All" if there is no dimensional analysis specified - `users` - the number of users in the variation-dimension that are part of the denominator for that metric; normally this is all users in the variation-dimension, but for ratio metrics or activation metrics it will only include users that are in the ratio denominator or in the set of activated users. - `count` - deprecated; duplicate of users - `statistic_type` - one of "mean" or "ratio"; this indicates to the statistics engine how to process the data as the formulae for the variance is different across these statistic types - `main_metric_type` - the metric type of the main metric (e.g. the metric itself or the numerator for ratio metrics) - `main_sum` - the summed value of the user-level main metric - `main_sum_squares` - the sum of the squared values of the user-level main metric If the metric is a ratio metric, you will additionally see the following fields: - `denominator_metric_type` - the metric type of the denominator metric - `denominator_sum` - the summed value of the user-level denominator metric - `denominator_sum_squares` - the sum of the squared values of the user-level denominator metric - `main_denominator_sum_product` - the sum of the product of the user-level main and denominator metric --- ## Contextual Bandit Technical Details import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Technical contextual bandit details Here we document the technical details behind GrowthBook's contextual bandit variation weight updates. ## Thompson Sampling Suppose we have $K$ variations, and we want to update the variation weights for a given context. For the $k$-th variation, define its mean as $\mu_k$, k = 1, ..., K. Denote the probabiity that $\mu_{k}$ is larger than the other means as as $w_k$. Under Thompson Sampling, $w_k$ is the variation weight for the $k$-th variation. Define the posterior mean of $\mu_k$ as $m_k$ and its posterior variance as $v_k^2$. Define its posterior standard deviation as $s_k = \sqrt{v_k^2}$. Define the posterior distribution of $\mu_k$ as $f_k(\mu_k)$. This best-arm probability can be expressed as: $$ \begin{align*} w_k &= P\left(\left[\mu_k > \mu_1 \right] \cap \left[\mu_k > \mu_2 \right] \cap \cdots \cap \left[\mu_k > \mu_{k-1} \right] \cap \left[\mu_k > \mu_{k+1} \right] \cap \cdots \cap \left[\mu_k > \mu_K \right]\right) \\ &= \int_{-\infty}^{\mu_{k}}\int_{-\infty}^{\mu_{k}} \cdots \int_{-\infty}^{\mu_{k}} \int_{-\infty}^{\infty} \int_{-\infty}^{\mu_{1}} \cdots \int_{-\infty}^{\mu_{1}} \prod_{i=1}^{K} f_{i}(\mu_i) d\mu_i \\ &= \int_{-\infty}^{\infty}f_{k}(\mu_k) \left\{\int_{-\infty}^{\mu_{k}} \cdots \int_{-\infty}^{\mu_{k}} \prod_{i\ne k} f_{i}(\mu_i) d\mu_i \right\} d\mu_k \\ &= \int_{-\infty}^{\infty}f_{k}(\mu_k) \prod_{i\ne k} \Phi\left(\frac{\mu_{k} - m_{i} }{s_{i}}\right) d\mu_k. \end{align*} $$ We leverage the fact that the posterior distributions of the $\mu_i$ are normal and independent. The result above can be expressed as the expected value of the function below over the distribution of $\mu_k$ $$ \prod_{i\ne k} \Phi\left(\frac{\mu_{k} - m_{i} }{s_{i}}\right). $$ We use Gauss-Hermite quadrature to approximate the integral. Our integral is well suited for Gauss-Hermite quadrature, as the function inside the integral is smooth and light-tailed. Let $$t = \frac{\mu_{k} - m_{k} }{s_{k}},$$ which implies that $$\mu_{k} = m_{k} + t s_{k}.$$ Define the standard normal distribution as $\phi(.)$. The integral can be represented as: $$ \begin{align*} w_k &= \int_{-\infty}^{\infty} \left\{ \prod_{i\ne k} \Phi\left(\frac{m_{k} + t s_{k} - m_{i} }{s_{i}}\right)\right\} \phi(t) dt. \end{align*} $$ Letting $u = t / \sqrt(2)$ places the integral in the standard form for the Gauss-Hermite quadrature, which is $$ \int_{-\infty}^{\infty} \exp^{-u^2} g(u) du. $$ --- ## Cuped Estimated Variance Technical Details # Technical CUPED details Here we document the technical details behind GrowthBook CUPED variance estimates. We begin with mean metrics, and then move to [ratio metrics](/statistics/cuped-technical#ratio-metric). ## Mean or Binomial Metric We use the notation below. We describe our approach in terms of revenue, but any Mean or Binomial metric can be substituted. 1. Define $Y_{C}$ $\left(Y_{T}\right)$ as the observed post-exposure revenue for a user exposed to control (treatment). 2. Define $X_{C}$ $\left(X_{T}\right)$ as the observed pre-exposure revenue for a user exposed to control (treatment). 3. Define $Y$ ($X$) as the post-exposure (pre-exposure) revenue for all users collectively in the experiment. 4. Define $\bar{Y}_{C}$ $\left(\bar{Y}_{T}\right)$ as the sample average post-exposure revenue for users exposed to control (treatment). 5. Define $\mu_{C}$ $\left(\mu_{T}\right)$ as the population average post-exposure revenue for users exposed to control (treatment). 6. Define $N_{C}$ $\left(N_{T}\right)$ as the number of users exposed to control (treatment). ## Mean or Binomial Metric, Absolute case For absolute inference, our target parameter is $$ \begin{align} \Delta_{A}&=\mu_{Y}-\mu_{YC}. \end{align} $$ As described in Equation 4 of ([Deng et al. 2013](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf)), we find the optimal $\theta$ using user data across both control and treatment: $$\theta = cov(Y, X) / var(X).$$ Our estimate of $\Delta_{A}$ is the difference in adjusted means $$ \begin{align} \hat{\Delta}_{A} &= \left(\bar{Y}_{T} - \theta\bar{X}_{T}\right) - \left(\bar{Y}_{C} - \theta\bar{X}_{C}\right). \end{align} $$ Under a superpopulation framework and independence of random assignment, the adjusted means $\left(\bar{Y}_{T} - \theta\bar{X}_{T}\right)$ and $\left(\bar{Y}_{C} - \theta\bar{X}_{C}\right)$ are statistically independent. Therefore, the variance of the difference in adjusted means is the sum of the variances of the adjusted means. We denote these variances as $V_{adj, C}$ and $V_{adj, T}$, respectively, and they are defined as Define the control (treatment) population covariance between post-exposure and pre-exposure revenue as $\sigma_{XY,C}$ ($\sigma_{XY,T}$). $$ \begin{align} V_{adj, C} &= \frac{\sigma^{2}_{YC} + \theta^{2}\sigma^{2}_{XC} - 2\theta\sigma_{XY,C}}{N_{C}}\\ V_{adj, T} &= \frac{\sigma^{2}_{YT} + \theta^{2}\sigma^{2}_{XT} - 2\theta\sigma_{XY,T}}{N_{T}}. \end{align} $$ Our estimated variance of $\hat{\Delta}_{A}$ is $\hat{\sigma}^{2}_{\Delta_{A}} = V_{adj, C} + V_{adj, T}$. Formulae for variances of the statistics can be found in our sections on [proportion metrics](/statistics/details#proportion-metrics) and [mean metrics](/statistics/details#mean-metrics). When estimating the covariance between two binomial variables $X$ and $Y$, we use the following formula: $$ \begin{align*} \sigma_{XY} &= E(XY) - E(X)E(Y) \\ &= P(X=1, Y=1) - P(X=1)P(Y=1) \\ &= P(X=1) * P(Y=1|X=1) - P(X=1)P(Y=1) \\ &= P(X=1) * (P(Y=1|X=1) - P(Y=1)). \end{align*} $$ For mean metrics, we use the usual covariance formula: $$ \begin{align*} \sigma_{XY} &= \frac{1}{N-1} \sum_{i=1}^{N} (X_i - \bar{X})(Y_i - \bar{Y}). \end{align*} $$ ## Mean or Binomial Metric, Relative case For relative inference (i.e., estimating lift), the parameter of interest is $$ \begin{align} \Delta_{R}&=\frac{\mu_{T}-\mu_{C}}{\mu_{C}}. \end{align} $$ Our estimate of $\Delta_{R}$ is the difference in adjusted means divided by the control mean: $$ \begin{align} \hat{\Delta}_{R} = \frac{\left(\bar{Y}_{T} - \theta\bar{X}_{T}\right) - \left(\bar{Y}_{C} - \theta\bar{X}_{C}\right)}{\bar{Y}_{C}}. \end{align} $$ To derive $\hat{\sigma}^{2}_{\Delta_{R}}$, the estimated variance of $\hat{\Delta}_{R}$, we use the delta method. 1. Define the control (treatment) population post-exposure variance as $\sigma^{2}_{YC}$ ($\sigma^{2}_{YT}$). 2. Define the control (treatment) population pre-exposure variance as $\sigma^{2}_{XC}$ ($\sigma^{2}_{XT}$). 3. Define the covariance of the sample control means $$ \boldsymbol{\Lambda}_{C} = \text{Cov}\left[\bar{Y}_{C}, \bar{X}_{C}\right] =\begin{pmatrix} \sigma^{2}_{Y,C} & \sigma*{XY,C}\\ \sigma*{XY,C} & \sigma^{2}_{X,C} \end{pmatrix}/ N_{C}$$. 4. Define the covariance of the sample treatment means $$ \boldsymbol{\Lambda}_{T} = \text{Cov}\left[\bar{Y}_{T}, \bar{X}_{T}\right] =\begin{pmatrix} \sigma^{2}_{Y,T} & \sigma*{XY,T}\\ \sigma*{XY,T} & \sigma^{2}_{X,T} \end{pmatrix}/ N_{T}$$. 5. Define the vector of population means $\boldsymbol{\beta}_{0} = \left[\mu_{YT}, \mu_{XT}, \mu_{YC}, \mu_{XC} \right].$ 6. Define their sample counterparts as $\hat{\boldsymbol{\beta}} = \left[\bar{Y}_{T}, \bar{X}_{T}, \bar{Y}_{C}, \bar{X}_{C} \right].$ 7. Define $$\boldsymbol{\Lambda} = \text{Cov}\left(\hat{\boldsymbol{\beta}}\right) = \begin{pmatrix} \boldsymbol{\Lambda}_{T} & \textbf{0}\\ \textbf{0} & \boldsymbol{\Lambda}_{C} \end{pmatrix},$$ where $\textbf{0}$ is a $2 \times 2$ matrix of zeros. By the multivariate central limit theorem: $$ \begin{align} \hat{\boldsymbol{\beta}} \stackrel{}{\sim}\mathcal{MVN}\left(\boldsymbol{\beta}_{0},\boldsymbol{\Lambda}\right). \end{align} $$ For vector $\boldsymbol{\beta}$, define its $k^{\text{th}}$ element as $\beta[k]$. Define the function $$g(\boldsymbol{\beta}; \theta) = \frac{\left(\beta[1] - \theta\beta[2]\right) - \left(\beta[3] - \theta\beta[4]\right)}{\beta[3]}.$$ Define the vector of partial derivatives as $\boldsymbol{\nabla}_{r} = \frac{\partial g(\boldsymbol{\beta})}{\partial\boldsymbol{\beta}}$, where the individual elements are $$ \begin{align*} \boldsymbol{\nabla}[1] &= \frac{1}{\boldsymbol{\beta}[3]} \\\boldsymbol{\nabla}[2] &= \frac{-\theta}{\boldsymbol{\beta}[3]} \\\boldsymbol{\nabla}[3] &= \frac{-\boldsymbol{\beta}[3] - \left(\boldsymbol{\beta}[1] - \theta\boldsymbol{\beta}[2] - \boldsymbol{\beta}[3] + \theta\boldsymbol{\beta}[4]\right) }{\boldsymbol{\beta}[3]^{2}} = \frac{ -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4] }{\boldsymbol{\beta}[3]^{2}} \\\boldsymbol{\nabla}[4] &= \frac{\theta}{\boldsymbol{\beta}[3]}. \end{align*} $$ By the delta method, $$\hat{\Delta}_{r} = g(\hat{\boldsymbol{\beta}}) \stackrel{}{\sim}\mathcal{N}\left(\Delta_{r} = g\left(\boldsymbol{\beta}\right), \boldsymbol{\nabla}_{r}^{\top}\Lambda\boldsymbol{\nabla}_{r} \right)$$. Decompose $\boldsymbol{\nabla}_{r}$ into $\boldsymbol{\nabla}_{r} = \left[ \boldsymbol{\nabla}_{r}[1:2], \boldsymbol{\nabla}_{r}[3:4] \right].$ Then the final variance $$ \begin{align*} \boldsymbol{\nabla}_{r}^{\top}\Lambda\boldsymbol{\nabla}_{r} &= \boldsymbol{\nabla}_{r}[1:2]^{\top} \boldsymbol{\Lambda}_{T} \boldsymbol{\nabla}_{r}[1:2] + \boldsymbol{\nabla}_{r}[3:4]^{\top} \boldsymbol{\Lambda}_{C} \boldsymbol{\nabla}_{r}[3:4] \nonumber\\&= \left[\frac{1}{\boldsymbol{\beta}[3]}, \frac{-\theta}{\boldsymbol{\beta}[3]}\right] ^{\top} \begin{pmatrix} \sigma^{2}_{Y,T} & \sigma_{XY,T} \\ \sigma_{XY,T} & \sigma^{2}_{X,T} \end{pmatrix}/ N_{T} \left[\frac{1}{\boldsymbol{\beta}[3]}, \frac{-\theta}{\boldsymbol{\beta}[3]}\right] \nonumber\\&+ \left[\frac{ -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4] }{\boldsymbol{\beta}[3]^{2}}, \frac{\theta}{\boldsymbol{\beta}[3]}\right] \begin{pmatrix} \sigma^{2}_{Y,C} & \sigma_{XY,C} \\ \sigma_{XY,C} & \sigma^{2}_{X,C} \end{pmatrix}/ N_{C} \left[\frac{ -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4] }{\boldsymbol{\beta}[3]^{2}}, \frac{\theta}{\boldsymbol{\beta}[3]}\right] \nonumber\\&= \frac{1}{N_{T}\boldsymbol{\beta}[3]^{2}}\left[1, -\theta\right] ^{\top} \begin{pmatrix} \sigma^{2}_{Y,T} & \sigma_{XY,T} \\ \sigma_{XY,T} & \sigma^{2}_{X,T} \end{pmatrix} \left[1, -\theta\right] \nonumber\\&+ \frac{1}{N_{C}\boldsymbol{\beta}[3]^{2}} \left[\frac{ -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4] }{\boldsymbol{\beta}[3]}, \theta\right] \begin{pmatrix} \sigma^{2}_{Y,C} & \sigma_{XY,C} \\ \sigma_{XY,C} & \sigma^{2}_{X,C} \end{pmatrix} \left[\frac{ -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4] }{\boldsymbol{\beta}[3]}, \theta\right] \nonumber\\&=\frac{ \sigma^{2}_{Y,T} +\theta^{2}\sigma^{2}_{X,T} -2\sigma_{XY,T} }{N_{T}\boldsymbol{\beta}[3]^{2}} \nonumber\\&+ \frac{1}{N_{C}\boldsymbol{\beta}[3]^{2}} \left[\frac{ -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4] }{\boldsymbol{\beta}[3]}, \theta\right] \begin{pmatrix} \frac{\sigma^{2}_{Y,C} \left( -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4] \right)}{ \boldsymbol{\beta}[3] } + \theta\sigma_{XY,C} \\ \frac{\sigma_{XY,C} \left( -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4] \right)}{\boldsymbol{\beta}[3]} +\theta\sigma^{2}_{X,C} \end{pmatrix} \nonumber\\&=\frac{ \sigma^{2}_{Y,T} +\theta^{2}\sigma^{2}_{X,T} -2\sigma_{XY,T} }{N_{T}\boldsymbol{\beta}[3]^{2}} \nonumber\\&+ \frac{1}{N_{C}\boldsymbol{\beta}[3]^{2}} \left( \frac{\sigma^{2}_{Y,C} \left( -\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4]\right)^{2}}{\boldsymbol{\beta}[3]^{2}} +2\frac{\theta\sigma_{XY,C} \left(-\boldsymbol{\beta}[1] + \theta\boldsymbol{\beta}[2] - \theta\boldsymbol{\beta}[4]\right)}{\boldsymbol{\beta}[3]} +\theta^{2}\sigma^{2}_{X,C} \right) \nonumber\\&=\frac{ \sigma^{2}_{Y,T} +\theta^{2}\sigma^{2}_{X,T} -2\sigma_{XY,T} }{N_{T}\bar{Y}_{C}^{2}} \nonumber\\&+ \frac{1}{N_{C}\bar{Y}_{C}^{2}} \left( \frac{\sigma^{2}_{Y,C} \left( -\bar{Y}_{T} + \theta\bar{X}_{T} - \theta\bar{X}_{C} \right)^{2}}{\bar{Y}_{C}^{2}} +2\frac{\theta\sigma_{XY,C} \left(-\bar{Y}_{T} + \theta\bar{X}_{T} - \theta\bar{X}_{C}\right)}{\bar{Y}_{C}} +\theta^{2}\sigma^{2}_{X,C} \right), \end{align*} $$ where in the last step we move away from $\boldsymbol{\beta}$ notation and use sample mean notation. For estimating uncertainty in production, we use $$ \begin{align} \hat{\sigma}^{2}_{\Delta_{R}}&=\frac{ \sigma^{2}_{Y,T} +\theta^{2}\sigma^{2}_{X,T} -2\sigma_{XY,T} }{N_{T}\bar{Y}_{C}^{2}} \nonumber\\&+ \frac{1}{N_{C}\bar{Y}_{C}^{2}} \left( \frac{\sigma^{2}_{Y,C} \left( -\bar{Y}_{T} \right)^{2}}{\bar{Y}_{C}^{2}} +2\frac{\theta\sigma_{XY,C} \left(-\bar{Y}_{T}\right)}{\bar{Y}_{C}} +\theta^{2}\sigma^{2}_{X,C} \right), \end{align} $$ which leverages the fact that the pre-exposure revenue population means are equal due to randomization. ## Ratio Metric Throughout define the $k^{\text{th}}$ element of vector $\textbf{x}$ as $\textbf{x}[k]$. Below we define parameters. 1. Under control, define the numerator (denominator) post-exposure population mean as $\mu_{MYC}$ ($\mu_{DYC}$). 2. Under treatment, define the numerator (denominator) post-exposure population mean as $\mu_{MYT}$ ($\mu_{DYT}$). 3. Under control, define the numerator (denominator) pre-exposure population mean as $\mu_{MXC}$ ($\mu_{DXC}$). 4. Under treatment, define the numerator (denominator) pre-exposure population mean as $\mu_{MXT}$ ($\mu_{DXT}$). Due to randomization, $\mu_{MXC}$ equals $\mu_{DXC}$ and $\mu_{MXT}$ $\mu_{DXT}$, but for bookkeeping purposes, it is easier to have separate parameters. ## Ratio Metric, Absolute case For ratio metrics the absolute parameter of interest is $$ \begin{align} \Delta_{a}&= \frac{\mu_{MYT}}{\mu_{DYT}} - \frac{\mu_{MYC}}{\mu_{DYC}}. \end{align} $$ Below we define statistics. 1. Under control, define the numerator (denominator) post-exposure sample mean as $\bar{M}_{YC}$ ($\bar{D}_{YC}$). 2. Under treatment, define the numerator (denominator) post-exposure sample mean as $\bar{M}_{YT}$ ($\bar{D}_{YT}$). 3. Under control, define the numerator (denominator) pre-exposure sample mean as $\bar{M}_{XC}$ ($\bar{D}_{XC}$). 4. Under treatment, define the numerator (denominator) pre-exposure sample mean as $\bar{M}_{XT}$ ($\bar{D}_{XT}$). Define: 1. $\boldsymbol{\beta}_{0} = \left[\mu_{MYT}, \mu_{DYT}, \mu_{MXT}, \mu_{DXT}, \mu_{MYC}, \mu_{DYC}, \mu_{MXC}, \mu_{DXC}\right]$. 2. $\hat{\boldsymbol{\beta}} = \left[\bar{M}_{YT}, \bar{D}_{YT}, \bar{M}_{XT}, \bar{D}_{XT}, \bar{M}_{YC}, \bar{D}_{YC}, \bar{M}_{XC}, \bar{D}_{XC} \right]$. 3. $\boldsymbol{\Lambda}_{T} = \text{Cov}\left[\bar{M}_{YT}, \bar{D}_{YT}, \bar{M}_{XT}, \bar{D}\_{XT},\right] N*{T}^{-1} \begin{pmatrix} \text{Var}\left(M_{YT}\right) & & &\\ \text{Cov}\left(M_{YT}, D_{YT}\right) & \text{Var}\left(D_{YT}\right) & & \\ \text{Cov}\left(M_{YT}, M_{XT}\right) & \text{Cov}\left(D_{YT}, M_{XT}\right) & \text{Var}\left(M_{XT}\right) & \\ \text{Cov}\left(M_{YT}, D_{XT}\right) & \text{Cov}\left(D_{YT}, D_{XT}\right) & \text{Cov}\left(M_{XT}, D_{XT}\right) & \text{Var}\left(D_{XT}\right) \\ \end{pmatrix}$ 4. $\boldsymbol{\Lambda}_{C} = \text{Cov}\left[\bar{M}_{YC}, \bar{D}_{YC}, \bar{M}_{XC}, \bar{D}\_{XC},\right] N*{C}^{-1} \begin{pmatrix} \text{Var}\left(M_{YC}\right) & & &\\ \text{Cov}\left(M_{YC}, D_{YC}\right) & \text{Var}\left(D_{YC}\right) & & \\ \text{Cov}\left(M_{YC}, M_{XC}\right) & \text{Cov}\left(D_{YC}, M_{XC}\right) & \text{Var}\left(M_{XC}\right) & \\ \text{Cov}\left(M_{YC}, D_{XC}\right) & \text{Cov}\left(D_{YC}, D_{XC}\right) & \text{Cov}\left(M_{XC}, D_{XC}\right) & \text{Var}\left(D_{XC}\right) \\ \end{pmatrix}$ 5. $\boldsymbol{\Lambda} = \text{Cov}\left(\hat{\boldsymbol{\beta}}\right) \begin{pmatrix} \boldsymbol{\Lambda}_{T} & \underset{4\times4}{\textbf{0}}\\ \underset{4\times4}{\textbf{0}} & \boldsymbol{\Lambda}_{C} \\ \end{pmatrix}$ Define the function $$ \begin{align} g\left(\boldsymbol{\beta}; \theta\right)_{a}&= \left(\frac{\boldsymbol{\beta}[1]}{\boldsymbol{\beta}[2]} - \theta\frac{\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]} \right) -\left(\frac{\boldsymbol{\beta}[5]}{\boldsymbol{\beta}[6]} - \theta\frac{\boldsymbol{\beta}[7]}{\boldsymbol{\beta}[8]} \right). \end{align} $$ The CUPED estimator is $$ \begin{align} \hat{\Delta}_{a}&=g_{a}(\hat{\boldsymbol{\beta}}; \theta) \\&= \left(\frac{\hat{\boldsymbol{\beta}}[1]}{\hat{\boldsymbol{\beta}}[2]} - \theta\frac{\hat{\boldsymbol{\beta}}[3]}{\hat{\boldsymbol{\beta}}[4]} \right) -\left(\frac{\hat{\boldsymbol{\beta}}[5]}{\hat{\boldsymbol{\beta}}[6]} - \theta\frac{\hat{\boldsymbol{\beta}}[7]}{\hat{\boldsymbol{\beta}}[8]} \right). \end{align} $$ Define the vector of partial derivatives as $\boldsymbol{\nabla}_{a}\left(\boldsymbol{\beta}; \theta\right) = \frac{\partial g_{a}(\boldsymbol{\beta})}{\partial\boldsymbol{\beta}}$. $$ \begin{align*} \boldsymbol{\nabla}[1] &= \frac{1}{\boldsymbol{\beta}[2]} \\\boldsymbol{\nabla}[2] &= \frac{-\boldsymbol{\beta}[1]}{\boldsymbol{\beta}[2]^2} \\\boldsymbol{\nabla}[3] &= \frac{-\theta}{\boldsymbol{\beta}[4]} \\\boldsymbol{\nabla}[4] &= \frac{\theta\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2} \\\boldsymbol{\nabla}[5] &= \frac{-1}{\boldsymbol{\beta}[6]} \\\boldsymbol{\nabla}[6] &= \frac{\boldsymbol{\beta}[5]}{\boldsymbol{\beta}[6]^2} \\\boldsymbol{\nabla}[7] &= \frac{\theta}{\boldsymbol{\beta}[8]} \\\boldsymbol{\nabla}[8] &= \frac{-\theta\boldsymbol{\beta}[7]}{\boldsymbol{\beta}[8]^2}. \end{align*} $$ Note that $\boldsymbol{\nabla}_{a}\left(\hat{\boldsymbol{\beta}}; \theta\right)$ $$ \begin{align*} \boldsymbol{\nabla}[1] &= \frac{1}{\bar{D}_{YT}} \\\boldsymbol{\nabla}[2] &= \frac{-\bar{M}_{YT}}{\bar{D}_{YT}^{2}} \\\boldsymbol{\nabla}[3] &= \frac{-\theta}{\bar{D}_{XT}} \\\boldsymbol{\nabla}[4] &= \frac{\theta\bar{M}_{XT}}{\bar{D}_{XT}^2} \\\boldsymbol{\nabla}[5] &= \frac{-1}{\bar{D}_{YC}} \\\boldsymbol{\nabla}[6] &= \frac{\bar{M}_{YC}}{\bar{D}_{YC}^2} \\\boldsymbol{\nabla}[7] &= \frac{\theta}{\bar{D}_{XC}} \\\boldsymbol{\nabla}[8] &= \frac{-\theta\bar{M}_{XC}}{\bar{D}_{XC}^2}. \end{align*} $$ By the central limit theorem, $$\hat{\Delta}_{a} = g(\hat{\boldsymbol{\beta}}) \stackrel{}{\sim}\mathcal{N}\left(\Delta_{r} = g\left(\boldsymbol{\beta}\right), \boldsymbol{\nabla}_{a}^{\top}\Lambda\boldsymbol{\nabla}_{a} \right)$$. Decompose $\boldsymbol{\nabla}_{a}$ into its first four elements ($\boldsymbol{\nabla}_{a, T}$) and its last four elements ($\boldsymbol{\nabla}_{a, C}$). Our variance of interest is $$ \begin{align} \hat{\sigma}^{2}_{\Delta_{a}}&= \boldsymbol{\nabla}_{a}\left(\hat{\boldsymbol{\beta}}; \theta\right)^{\top}\boldsymbol{\Lambda}\boldsymbol{\nabla}_{a}\left(\hat{\boldsymbol{\beta}}; \theta\right) \nonumber\\ &= \boldsymbol{\nabla}_{a, T}^{\top}\boldsymbol{\Lambda}_{T}\boldsymbol{\nabla}_{a, T} + \boldsymbol{\nabla}_{a, C}^{\top}\boldsymbol{\Lambda}_{C}\boldsymbol{\nabla}_{a, C}. \end{align} $$ All of these moments are available via CupedRatioRegressionAdjustedStatistics. ## Optimal regression coefficient for ratio metrics The optimal $\theta$ minimizes Equation (14). We can write $$ \begin{align*} \boldsymbol{\nabla}_{a, T}^{\top}\boldsymbol{\Lambda}_{T}\boldsymbol{\nabla}_{a, T}&=\boldsymbol{\nabla}_{a, T}[1:2]^{\top}\boldsymbol{\Lambda}_{T}[1:2, 1:2]\boldsymbol{\nabla}_{a, T}[1:2] \nonumber\\&+2\boldsymbol{\nabla}_{a, T}[1:2]^{\top}\boldsymbol{\Lambda}_{T}[3:4, 1:2]\boldsymbol{\nabla}_{a, T}[3:4]+\boldsymbol{\nabla}_{a, T}[3:4]^{\top}\boldsymbol{\Lambda}_{T}[3:4, 3:4]\boldsymbol{\nabla}_{a, T}[3:4] \nonumber\\ &=c_{T} + 2\theta \left[\frac{1}{\boldsymbol{\beta}[2]}, \frac{-\boldsymbol{\beta}[1]}{\boldsymbol{\beta}[2]^2} \right]^{\top} \boldsymbol{\Lambda}_{T}[3:4, 1:2]\left[\frac{-1}{\boldsymbol{\beta}[4]}, \frac{\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2} \right] + \theta^{2}\left[\frac{-1}{\boldsymbol{\beta}[4]}, \frac{\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2}\right]^{\top}\boldsymbol{\Lambda}_{T}[3:4, 3:4]\left[\frac{-1}{\boldsymbol{\beta}[4]}, \frac{\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2}\right] \end{align*} $$ where $c_{T}$ is free of $\theta$. Similarly we can write $$ \begin{align*} \boldsymbol{\nabla}_{a, C}^{\top}\boldsymbol{\Lambda}_{C}\boldsymbol{\nabla}_{a, C}&= c_{C} + 2\theta \left[\frac{-1}{\boldsymbol{\beta}[2]}, \frac{\boldsymbol{\beta}[1]}{\boldsymbol{\beta}[2]^2} \right]^{\top} \boldsymbol{\Lambda}_{C}[3:4, 1:2]\left[\frac{1}{\boldsymbol{\beta}[4]}, \frac{-\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2} \right] + \theta^{2}\left[\frac{-1}{\boldsymbol{\beta}[8]}, \frac{\boldsymbol{\beta}[7]}{\boldsymbol{\beta}[8]^2}\right]^{\top}\boldsymbol{\Lambda}_{C}[3:4, 3:4]\left[\frac{-1}{\boldsymbol{\beta}[8]}, \frac{\boldsymbol{\beta}[7]}{\boldsymbol{\beta}[8]^2}\right] \end{align*} $$ where $c_{C}$ is a constant free of $\theta$. Differentiating the sum of these two equations with respect to $\theta$ and setting equal to zero shows that the minimum of this quadratic form occurs at $$ \begin{align} \theta_{\text{opt}} &= -\frac{ \left[\frac{-1}{\boldsymbol{\beta}[2]}, \frac{\boldsymbol{\beta}[1]}{\boldsymbol{\beta}[2]^2} \right]^{\top} \boldsymbol{\Lambda}_{C}[3:4, 1:2]\left[\frac{1}{\boldsymbol{\beta}[4]}, \frac{-\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2} \right] + \left[\frac{1}{\boldsymbol{\beta}[2]}, \frac{-\boldsymbol{\beta}[1]}{\boldsymbol{\beta}[2]^2} \right]^{\top} \boldsymbol{\Lambda}_{T}[3:4, 1:2]\left[\frac{-1}{\boldsymbol{\beta}[4]}, \frac{\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2} \right] }{\left[\frac{-1}{\boldsymbol{\beta}[8]}, \frac{\boldsymbol{\beta}[7]}{\boldsymbol{\beta}[8]^2}\right]^{\top}\boldsymbol{\Lambda}_{C}[3:4, 3:4]\left[\frac{-1}{\boldsymbol{\beta}[8]}, \frac{\boldsymbol{\beta}[7]}{\boldsymbol{\beta}[8]^2}\right] + \left[\frac{-1}{\boldsymbol{\beta}[4]}, \frac{\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2}\right]^{\top}\boldsymbol{\Lambda}_{T}[3:4, 3:4]\left[\frac{-1}{\boldsymbol{\beta}[4]}, \frac{\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2}\right] }. \end{align} $$ The numerator represents the sum of the covariances, the denominator the sum of the variances. This is the same $\theta$ as is presented in Appendix B of ([Deng et al. 2013](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf)). ## Ratio Metric, Relative case For relative inference much of the approach in the previous section works, we simply need to define the appropriate $g$ function and its partial derivatives. For ratio metrics the relative parameter of interest is $$ \begin{align} \Delta_{r}&=\frac{\frac{\mu_{MTY}}{\mu_{DTY}} - \frac{\mu_{MCY}}{\mu_{DCY}}}{\frac{\mu_{MCY}}{\mu_{DCY}}} \\&= \frac{\frac{\mu_{MTY}}{\mu_{DTY}}}{\frac{\mu_{MCY}}{\mu_{DCY}}} - 1. \end{align} $$ Define the function $$ \begin{align} g\left(\boldsymbol{\beta}; \theta\right)_{r}&= \frac{ \left(\frac{\boldsymbol{\beta}[1]}{\boldsymbol{\beta}[2]} - \theta\frac{\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]} \right) -\left(\frac{\boldsymbol{\beta}[5]}{\boldsymbol{\beta}[6]} - \theta\frac{\boldsymbol{\beta}[7]}{\boldsymbol{\beta}[8]} \right) }{\frac{\boldsymbol{\beta}[5]}{\boldsymbol{\beta}[6]}} \end{align} $$ We can consistently estimate Equation (18) with the CUPED estimator $$g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r}.$$ Define the numerator in Equation (18) as $g\left(\boldsymbol{\beta}; \theta\right)_{r, num}$ and the denominator as $g\left(\boldsymbol{\beta}; \theta\right)_{r, den}$. Define the vector of partial derivatives as $\boldsymbol{\nabla}_{r}\left(\boldsymbol{\beta}; \theta\right) = \frac{\partial g_{r}(\boldsymbol{\beta})}{\partial\boldsymbol{\beta}}$. $$ \begin{align*} \boldsymbol{\nabla}[1] &= \frac{\frac{1}{\boldsymbol{\beta}[2]} }{g\left(\boldsymbol{\beta}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[2] &= \frac{\frac{-\boldsymbol{\beta}[1]}{\boldsymbol{\beta}[2]^2}}{g\left(\boldsymbol{\beta}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[3] &= \frac{ \frac{-\theta}{\boldsymbol{\beta}[4]} }{g\left(\boldsymbol{\beta}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[4] &= \frac{\frac{\theta\boldsymbol{\beta}[3]}{\boldsymbol{\beta}[4]^2}}{g\left(\boldsymbol{\beta}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[5] &= \frac{ \frac{-g\left(\boldsymbol{\beta}; \theta\right)_{r, den}}{\boldsymbol{\beta}[6]} - \frac{g\left(\boldsymbol{\beta}; \theta\right)_{r, num}}{\boldsymbol{\beta}[6]} }{g\left(\boldsymbol{\beta}; \theta\right)_{r, den}^{2}} \\\boldsymbol{\nabla}[6] &= \frac{ \frac{\boldsymbol{\beta}[5]g\left(\boldsymbol{\beta}; \theta\right)_{r, den}}{\boldsymbol{\beta}[6]^2} + \frac{\boldsymbol{\beta}[5]g\left(\boldsymbol{\beta}; \theta\right)_{r, num}}{\boldsymbol{\beta}[6]^2} }{g\left(\boldsymbol{\beta}; \theta\right)_{r, den}^{2}} \\\boldsymbol{\nabla}[7] &= \frac{\frac{\theta}{\boldsymbol{\beta}[8]} }{g\left(\boldsymbol{\beta}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[8] &= \frac{ \frac{-\theta\boldsymbol{\beta}[7]}{\boldsymbol{\beta}[8]^2} }{g\left(\boldsymbol{\beta}; \theta\right)_{r, den}}. \end{align*} $$ Note that $$ \begin{align*} g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, num} &= \left(\frac{\bar{M}_{YT}}{\bar{D}_{YT}} - \theta\frac{\bar{M}_{XT}}{\bar{D}_{XT}} \right) -\left(\frac{\bar{M}_{YC}}{\bar{D}_{YC}} - \theta\frac{\bar{M}_{XC}}{\bar{D}_{XC}} \right) \\g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den} &= \frac{\bar{M}_{YC}}{\bar{D}_{YC}} \end{align*} $$ Note that $\boldsymbol{\nabla}_{r}\left(\hat{\boldsymbol{\beta}}; \theta\right)$ is equal to $$ \begin{align*} \boldsymbol{\nabla}[1] &= \frac{\frac{1}{\bar{D}_{YT}}}{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[2] &= \frac{\frac{-\bar{M}_{YT}}{\bar{D}_{YT}^{2}}}{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[3] &= \frac{ \frac{-\theta}{\bar{D}_{XT}} }{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[4] &= \frac{\frac{\theta\bar{M}_{XT}}{\bar{D}_{XT}^2}}{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[5] &= \frac{ \frac{-g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}}{\bar{D}_{YC}} - \frac{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, num}}{\bar{D}_{YC}} }{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}^{2}} \\\boldsymbol{\nabla}[6] &= \frac{ \frac{\bar{M}_{YC}g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}}{\bar{D}_{YC}^2} + \frac{\bar{M}_{YC}g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, num}}{\bar{D}_{YC}^2} }{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}^{2}} \\\boldsymbol{\nabla}[7] &= \frac{\frac{\theta}{\bar{D}_{XC}} }{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}} \\\boldsymbol{\nabla}[8] &= \frac{ \frac{-\theta\bar{M}_{XC}}{\bar{D}_{XC}^2}. }{g\left(\hat{\boldsymbol{\beta}}; \theta\right)_{r, den}}. \end{align*} $$ --- ## CUPED import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; # CUPED + post-stratification CUPEDps (Controlled-experiment Using Pre-Experiment Data + post-stratification) increases the velocity of experimentation by reducing the uncertainty in estimates of experiment uplift. It uses both pre-experiment metric data as well as user attributes to improve the accuracy of experiment results. ## Why use CUPEDps? CUPEDps decreases the variance of experiment uplift, increasing the accuracy of your experimental results and therefore the speed at which you can see the effects of an experiment. In the right conditions, CUPEDps can equate to getting 20% or more traffic during your experiment! - In 2016, Netflix reported that CUPED reduced variance by roughly ~40% for some key engagement metrics ([source](https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf)). - In 2022, Microsoft reported that, for one product team, CUPED was akin to adding 20% more traffic to analysis of a majority of metrics ([source](https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/deep-dive-into-variance-reduction/)). ### How does it work? CUPEDps leverages both regression adjustment and post-stratification to improve the accuracy of your experiment results. Suppose revenue is your key metric, and you have historical revenue data on your users. You know that revenue spend varies across country (e.g. US vs. UK), and historical revenue spend is correlated with future revenue spend. In an ideal experiment, you would: 1) bucket users by country; 2) create pairs of users within each bucket that have the same revenue spend; and 3) randomly assign one user in each pair to the treatment and the other to the control. Balancing helps ensure the treatment and control groups are similar before the experiment starts, and makes it easier to detect post-randomization differences. However, this level of assignment in online experimentation is often difficult to achieve. CUPEDps is a statistical technique that permits you to answer the question, "If I had run the ideal experiment, what would the lift be?" Regression adjustment helps balance users with respect to their pre-randomization revenue spend. Post-stratification helps balance users by country and account for spending differences across countries. The more correlated the pre-experiment metric data and attributes are with your metric of interest, the more variance reduction you can achieve. For example, the following plot demonstrates the difference in the distribution of a metric before adjustment ("Normal") and after adjustment ("Adjusted"). In both panels, the green, adjusted metric is distributed less widely (e.g. it is more tightly spaced out around the mean). However, the adjusted distribution is even tighter in the right plot, showing that variance reduction will be greater the more correlated your pre-experiment data is with your post-experiment data. ![Variance Reduction by Correlation](/images/statistics/cuped-corr.png) In simpler terms, if we know a particular user tends to buy a lot of products on your website before you launch an experiment, or if a certain type of users tends to have high spend, we can use that information to understand whether purchase behavior after an experiment is driven by that customer's innate behavior or whether you can attribute it to the experiment. The concept of regression adjustment has been around for a long time, but you should feel free to read more in the original CUPED paper ([Deng et al. 2013](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf)), a more general purpose paper on the underpinnings of regression adjustment from a sampling perspective ([Lin 2013](https://projecteuclid.org/journals/annals-of-applied-statistics/volume-7/issue-1/Agnostic-notes-on-regression-adjustments-to-experimental-data--Reexamining/10.1214/12-AOAS583.full)), and any of the many blog posts on the topic (e.g. [Booking.com](https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d), [Microsoft](https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/deep-dive-into-variance-reduction/)). ### Why CUPEDps is different We have combined CUPED with post-stratification to bring additional accuracy gains in three ways. Most importantly, we estimate average lift across all users by weighting the strata-specific (e.g. country-specific) lifts. Leveraging within-strata lifts isolates the treatment effect from the often substantial between-strata noise (e.g., the large natural differences in revenue between countries). This can significantly increase the precision of the overall lift estimate. For example, suppose average user revenue is high in the US and UK, but low in France. A two-sample t-test simply compares the averages of the two groups, ignoring strata information. In contrast, CUPEDps estimates are weighted averages of the lift for US users, the lift for UK users, and the lift for French users. Each of these three apples-to-apples comparisons are unaffected by the between-strata variability, and appropriately weighting them provides our lift estimate. Second, if there is an imbalance in the observed proportion of treatment users in a country compared to the variation split, then post-stratification will adjust the lift estimate for that country to account for the imbalance. For example, if the desired variation split for an experiment is 50/50, but within France it is 48/52, post-stratification will upweight the French control user metrics and downweight their treatment counterparts to reflect the true variation split. Finally, we can estimate country-specific lifts by running CUPED for each country individually. This can lead to additional variance reduction when the country-specific relationships in pre-experiment and post-exposure metric behavior vary. You can read more technical details about when post-stratification is useful and how, with large enough data, it is often as good as stratifying your experiment ahead of time (see section 3.4 in [Xie and Aurisset 2016](https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf)). ## Configuring CUPEDps ### Organization-level settings You can turn CUPEDps on for all analyses under Settings → General. CUPEDps can be turned on or off by default for all analyses and you can set the default number of days to use for a lookback window. This setting will set the default for all of your metrics, which will then flow through to all analyses that use those metrics. ![Organization-level CUPED Setting](/images/statistics/cuped-org.png) Additionally, you can choose whether to enable post-stratification in the same location in your organization settings. For post-stratification to take effect, you **must** enable pre-computed dimensions in your organization settings as well as have at least one dimension configured in your experiment assignment query. ### Metric-level settings You can also override these organization-level defaults at the Metric level. When creating or editing a Metric, go to the "Behavior" panel, click "Show advanced options" and scroll to the bottom. From there, you will see the following settings. These settings will allow you to disable CUPED for a metric, even if it is set at the organization or experiment level. ![Metric-level CUPED Setting](/images/statistics/cuped-metric.png) You might want to disable CUPEDps for a particular metric if that metric never collects values for a user before they enter an experiment. You might want to adjust metric-specific lookback windows for any of the reasons listed in the section above. Finally, you could also, if you wanted, override these metric-level settings for a particular experiment using metric overrides on the experiment page. ### Experiment settings and results By default, each experiment will use your organization-level defaults, unless they are overridden by a metric. However, you can always toggle CUPED on or off using the toggle added to the top of the results table. You need to re-run your analysis if you change this setting. If the toggle is On, then CUPEDps will be applied for all metrics, excluding the cases described above (e.g., legacy ratio metrics) and exceptions like metric overrides. The following screenshot shows the results with CUPEDps on. However, you can see that there is an icon showing that CUPEDps is disabled for revenue_p50 (because it is a quantile metric). So even if CUPEDps is toggled on, we will always show you any metrics for which GrowthBook did not use CUPEDps using that small CUPEDps icon with a red x. ![Experiment Results with CUPED toggle](/images/statistics/cuped-results-ratio.png) The screenshot below shows the same data with CUPEDps toggled off. Note that the interval widths for the CUPEDps-enabled metrics are now much bigger, while the interval width for revenue_p50 is the same. CUPEDps increases the accuracy of your estimates. ![Experiment Results with CUPED toggle](/images/statistics/cuped-results-ratio-off.png) ### Regression Adjustment lookback window GrowthBook defaults to using 14 days of pre-exposure data, but this is customizable at the organization, metric, and metric-experiment level. Why use a longer lookback window? - A longer window can be better if your metric is low frequency and a longer window is needed to capture meaningful user behavior that will be correlated across the pre- and post-exposure time periods Why use a shorter lookback window? - Shorter lookback windows will yield more performant queries (fewer days to scan from your metric source) - Behavior in the days just before a user enters an experiment is likely to be more highly correlated with behavior during an experiment, as users change over time. Of course, this is mostly true for metrics that are observed at a higher frequency (e.g. simple engagement metrics). ### Availability CUPEDps works for all metrics except for: - Ratio metrics that are the goal metric for a [Bandit](/bandits/overview) - Quantile metrics - Legacy (non-fact) ratio metrics - Legacy (non-fact)metrics with custom user value aggregations - Metrics from a MixPanel data sources ## CUPEDps implementation GrowthBook takes a transparent, simple approach to regression adjustment. For each metric you analyze, we use the metric itself from the pre-exposure period as the correlated data. This tends to be very powerful for metrics that are frequently produced by users (e.g. engagement measures), but can be less powerful if your metric is rare, or if you are measuring behavior for new users. In general, CUPED is more powerful the more you know about your units of interest and the longer they have been able to generate the metric that you are analyzing as a part of your experiment. We then use the standard CUPED estimator for each variation mean, $$ \bar{Y}_{adjusted} = \bar{Y} - \theta * \bar{X} $$ where $\bar{Y}$ is the post-exposure metric average, $\bar{X}$ is the pre-exposure metric average, and $\theta$ is essentially a regression coefficient from a regression of the post-experiment data on the pre-experiment data (pooled across both the control and treatment variation of interest), $\theta = \text{Cov}(Y, X) / \text{Var}(X)$. We describe CUPED for ratio metrics and how to estimate uncertainty [here](/statistics/cuped-technical). As discussed above, we could use any correlated data instead of $X$. For example, we could use some model that includes all pre-exposure metrics added to your experiment, or auxiliary dimension information you have configured per user. However, one downside of these approaches is that your results for metric A will depend on whether or not you add a metric B to your experiment and our analysis pipeline would lose its modularity, where each metric can be processed in parallel. To resolve this, we have added post-stratification to CUPED, creating CUPEDps, which allows you to use key user dimensions to improve variance reduction beyond vanilla CUPED. We describe technical details of post-stratification [here](/statistics/post-stratification). --- ## Statistical Details # Statistical Details Both our Bayesian and our Frequentist engines begin with a similar foundation for estimating experiment effects. We estimate the experiment effect (either the **relative** lift or the **absolute** effect), and its standard error. We create decision making tools from those estimates (e.g., frequentist confidence intervals and p-values, or Bayesian credible intervals and risks) to help you make rollout/rollback decisions. Our estimates compare an experimental variation (henceforth the treatment) to some baseline variation (henceforth control). Define $\mu_{C}$ as the population control mean, and define $\mu_{T}$ as the population treatment mean. The absolute treatment effect is $\Delta_{a} = \mu_{T}-\mu_{C}$. The relative treatment effect (or lift) is $\Delta_{r} = (\mu_{T}-\mu_{C})/\mu_{C}$ if $\mu_{C}\ne 0$ and is undefined otherwise. Throughout for any population parameter $\gamma$ denote its sample counterpart as $\hat{\gamma}$. For example, the sample absolute effect is $\hat{\Delta}_{a}$. ### Lift (Relative Effects) By default, we estimate lift (i.e. relative effects or percentage changes) from the control to the treatment variations. No matter which engine we use, the statistics we leverage are $$ \begin{align} \hat{\Delta}_r &= \frac{\hat\mu_T - \hat\mu_C}{\hat\mu_C} \\ \hat{\sigma}^2_{\Delta_r} &= \frac{\hat\sigma^2_C \hat\mu^2_T}{\hat\mu^4_C n_C} + \frac{\hat\sigma^2_T}{\hat\mu^2_C n_T} \end{align} $$ where $\hat\mu_C$ and $\hat\mu_T$ are the estimates of our variation means, $\hat\sigma^2_C$ and $\hat\sigma^2_T$ are the estimated variances of those means, and $n_C$ and $n_T$ are the sample sizes. The variance $\hat{\sigma}^2_{\Delta_r}$ is a delta method estimator, as $\hat{\Delta}_r$ is a ratio. We cover how we estimate the variation means and their standard errors below, depending on metric type. ### Absolute Effects The math for absolute effects is simpler, as our estimator is no longer a ratio. $$ \begin{align} \hat{\Delta}_a &= \hat\mu_T - \hat\mu_C \\ \hat{\sigma}^2_{\Delta_a} &= \frac{\hat\sigma^2_C}{n_C} + \frac{\hat\sigma^2_T}{n_T} \end{align} $$ ## Bayesian Engine Our Bayesian engine synthesizes the above estimates with information external to the experiment to estimate lift. This synthesis combines the experimental data with the prior distribution, which contains information about the treatment effect before the experiment began. We specify the following prior $$ \Delta_{prior} \sim N(\mu_{prior}, \sigma_{prior}^{2}). $$ This information is represented by the prior mean $\mu_{prior}$ and the prior variance $\sigma_{prior}^{2}$. The prior mean is your best guess for the treatment effect before the experiment starts. The prior variance determines your confidence in this best guess. A small prior variance corresponds to high confidence, and vice versa. GrowthBook's default prior is an improper prior (i.e., $\sigma_{\text{prior}}^{2}=\infty$) that has no impact. As of GrowthBook 3.0, you can specify a prior that overrides the default. As stated above, the prior distribution represents your knowledge of the treatment effect before the experiment begins. GrowthBook uses priors on lift, as this is often easier to conceptualize (e.g., 95% chance the true lift is between -50% and 50%). This knowledge can be weak or strong, and we outline a few examples below. 1. Weak knowledge: suppose you have little information about your treatment effect, and do not have past data about treatment effects or experiments for this metric. Then use a weak prior, with mean 0 and large variance (e.g., $0.5^ 2$ or 1). 2. Moderate knowledge: perhaps you have run multiple experiments on this metric. Suppose the average lift for these experiments was 0.01, and the variance of lifts was 0.05. Then a prior with mean 0.01 and variance 0.05 can be appropriate. As another example, suppose you believe that your feature impact will be relatively moderate. A $N(0, 0.3^2)$ prior, our default when proper priors are enabled, encodes the prior belief that 68% of all experiments have a lift between -30% and 30%, and 95% of all experiments have a lift between -60% and 60%. 3. Strong knowledge: suppose you ran a similar experiment last year on the same feature, or you ran this experiment last quarter on a different segment, and your treatment effect estimate was 0.02 and its variance was 0.01. Then a prior with mean 0.02 and variance 0.01 can be appropriate. In summary, picking the right prior can add information to your results. If you use a moderately informative or strongly informative prior, conduct a sensitivity analysis by comparing your results to those using a weak prior, to see how sensitive your results are to prior selection. For any proper prior, the effect of the prior diminishes as the sample size increases. These priors are normally distributed and our effect estimates above are asymptotically normally distributed via the Central Limit Theorem. Therefore, combining them to compute our posterior beliefs, which will themselves be normally distributed, we get the following mean and variance for our posterior effect estimates: $$ \begin{align} \Delta_{posterior} &= \frac{ \frac{\mu_{prior}}{\sigma_{prior}^{2}} + \frac{\hat{\Delta}}{\hat{\sigma}^2_{\Delta} }}{ \frac{1}{\sigma^2_{prior}} + \frac{1}{\hat{\sigma}^2_{\Delta}}} \\ \sigma^2_{posterior} &= \frac{1}{\frac{1}{\sigma^2_{prior}} + \frac{1}{\hat{\sigma}^2_{\Delta}}} \end{align} $$ For relative effects, we simply plug in the values for our prior and the $\hat\Delta_r$ and $\hat{\sigma}^2_{\Delta_r}$ values from equations (1) and (2). For absolute effects, we first rescale the prior so that your prior beliefs represent the same amount of uncertainty for both relative and absolute effects. So we recompute your prior as the following: $$ \begin{align*} \mu_{prior,a} &= \mu_{prior} \left|\hat\mu_C\right| \\ \sigma^2_{prior,a} &= \sigma^2_{prior} \hat\mu^2_C \\ \end{align*} $$ where $\mu_{prior, a}$ is the prior mean and $\sigma_{prior,a}^{2}$ is the prior variance on the absolute scale. From the posterior, we can compute the following quantities of interest ### Chance To Win Chance to Win is the percentage of the posterior that is greater than 0 in favor of the treatment variation $$ CTW = 100\% * (1 - \Phi_{p osterior}(0)), $$ where $\Phi_{posterior}$ is the CDF of the distribution $N(\Delta_{posterior}, \sigma^2_{posterior})$. ### Risk Note: we no longer use risk in the GrowthBook app, but still compute it in the experiment results database. Risk is the expected loss, where loss is considered to be 0 if the variation is beneficial. Define $p$ as the probability that treatment is worse than the control (i.e., 1 - Chance to Win). Define $L$ as the average loss under the scenario where treatment is worse than control. Then the risk is $p \cdot L$. We model the loss $L$ as the mean of a truncated normal distribution with mean $\Delta_{posterior}$, variance $\sigma^2_{posterior}$, and boundaries at $-\infty$ and 0 for a variation, and 0 and $\infty$ for baseline. ### Confidence Interval Our “confidence interval” in the Bayesian engine is an interval from the 2.5th to the 97.5th percentile of the posterior distribution (e.g. $\Phi^{-1}_{posterior}(0.025)$ and $\Phi^{-1}_{posterior}(0.975)$). We plot the posterior between these two points in the GrowthBook UI. ## Frequentist Engine In our frequentist engine, we directly use $\hat\Delta_a$, $\hat\Delta_r$, and their standard errors. **Sequential Testing** - if you have sequential testing enabled, we implement Asymptotic Confidence Sequences, which you can read more about in the [sequential testing documentation](/statistics/sequential). Enabling sequential testing does not affect the mean $\hat\Delta$, but it inflates the standard error. ### p-value The p-value is the probability of observing the value $\hat{\Delta}/\hat{\sigma}_{\Delta_{a}}$if the true $\Delta$ was zero. We conduct two-tailed tests, so the p-value if $$ p = 2\left(1 - F_t\left(\left|\frac{\hat\Delta}{\hat\sigma_\Delta}\right|, \nu\right)\right) $$ where $F_t$ is the CDF t-distribution with degrees of freedom $\nu$ estimated via the Welch-Satterthwaite approximation. This converges to using the Normal distribution as sample size increases. ### Confidence Interval We return 95% confidence intervals. They are $$ \left[\hat\Delta - F^{-1}_t\left(0.975, \nu\right) \hat\sigma_\Delta,\; \hat\Delta + F^{-1}_t\left(0.975, \nu\right) \hat\sigma_\Delta\right] $$ ## Estimating variation means Our estimates of variation means and their variances ($\mu_C$, $\mu_T$, $\sigma^2_C$, and $\sigma^2_T$) are the same for both engines. In the following, we will focus on the control variation for simplicity. The math is the same for the treatment variation. While there is no difference across engines, there is a difference in our estimates depending on the metric type being analyzed. ### Mean metrics For mean metrics (e.g. the average revenue per user) we use standard sample mean estimators. This is used for: - Metrics that are of type `revenue`, `duration`, and `count` metrics and do not have denominators - or, Fact Metrics of type `mean` In these cases, we have, for both variations $$ \begin{align} \hat\mu_C &= \frac{\sum^{n_C}_{i=1} {Y_{i}}}{n_C} \\ \hat\sigma^2_C &= \frac{1}{n_C - 1}\left(\sum^{n_C}_{i=1} Y^2_i - \frac{\left(\sum^{n_C}_{i=1} Y_i\right)^2}{n_C}\right) \end{align} $$ where $Y_i$ is the $i^{\text{th}}$ unit in the control variation's total metric value. ### Proportion metrics Proportion metrics (e.g. the % of users who purchased a product) cover the following cases: - regular Metrics of type `binomial` - Fact Metrics of type `proportion` In these cases, we have $$ \begin{align} \hat\mu_C &= \frac{\sum^{n_C}_{i=1} {Y_{i}}}{n_C} \\ \hat\sigma^2_C &= \hat\mu_C (1 - \hat\mu_C) \end{align} $$ where $Y_i$ is either 0 or 1 for a user. ### Ratio metrics Ratio metrics (e.g. the bounce rate for the number of bounced sessions over the number of total session) require a bit more care as the unit of analysis (e.g. the session) is not the same as the unit of randomization (e.g. the user). Ratio metrics in GrowthBook are: - regular Metrics with a denominator that is type `revenue`, `duration`, and `count` - Fact Metrics of type `ratio` In these cases, we have $$ \begin{align} \hat\mu_C &= \frac{\sum^{n_C}_{i=1} {M_{i}}}{\sum^{n_C}_{i=1} {D_{i}}} \\ \hat\sigma^2_C &= \frac{1}{\hat\mu^2_D}\left(\hat\sigma^2_M - 2 \frac{\hat\mu_M}{\hat\mu_D}\hat\sigma_{MD} + \hat\sigma^2_D\frac{\hat\mu^2_M}{\hat\mu^2_D} \right) \end{align} $$ where $M_i$ and $D_i$ are the $i^{\text{th}}$ units' values for the numerator and denominator of the metric, $\hat\mu_M$ and $\hat\sigma^2_M$ are the estimated sample mean and variance of that metric, and $\hat\sigma_{MD}$ is the estimated covariance of M and D in the control variation. ### Quantile metrics The statistics for quantile metrics are covered more in detail in the [Quantile documentation](/statistics/quantile). But in the end we arrive at both a $\hat\mu$ and $\hat\sigma^2$ for the desired quantile and its variance and use those in our lift calculations. ## References For further reading on the statistical methods referenced throughout this document, see the sources below: - [Delta Method – Wikipedia](https://en.wikipedia.org/wiki/Delta_method) - [Effect Size: Difference in Means – Wikipedia](https://en.wikipedia.org/wiki/Effect_size) - [Bayesian Inference – Wikipedia](https://en.wikipedia.org/wiki/Bayesian_inference) - [Posterior Predictive Distribution – Wikipedia](https://en.wikipedia.org/wiki/Posterior_predictive_distribution) - [Truncated Normal Distribution – Wikipedia](https://en.wikipedia.org/wiki/Truncated_normal_distribution) - [Credible Interval – Wikipedia](https://en.wikipedia.org/wiki/Credible_interval) - [P-value – Wikipedia](https://en.wikipedia.org/wiki/P-value) - [Welch–Satterthwaite Equation – Wikipedia](https://en.wikipedia.org/wiki/Welch%E2%80%93Satterthwaite_equation) - [Confidence Interval – Wikipedia](https://en.wikipedia.org/wiki/Confidence_interval) - [Sample Mean and Covariance – Wikipedia](https://en.wikipedia.org/wiki/Sample_mean_and_covariance) --- ## Multiple Testing Corrections import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Multiple Testing Corrections :::info Multiple testing corrections are only implemented for the Frequentist statistics engine. ::: If you test a bunch of things at once, you will naturally get some results that look good, but aren't real! This page explains this problem in detail and how you can control it with GrowthBook. ## What is the multiple testing problem? In the frequentist framework, testing more than one hypothesis at a time increases the probability that you find a false positive, beyond the user's specified rate of $\alpha$, often 0.05. This is often known as the [multiple testing](https://en.wikipedia.org/wiki/Multiple_comparisons_problem), or multiple comparisons, problem. In online AB testing, experimenters will often be running hundreds of tests, if not tens of thousands, of tests, at the same time. In this setting, a "test" tends to be the comparison of variation A vs variation B, for one group, for one metric. If you are running 10 experiments with 2 variations, each with 10 metrics, you are running 100 tests at one time. Even if the test had no effect on any metric, having this many tests will dramatically increase the chance of seeing a false positive. ### Error rates To understand approaches to solving this issue, let's first define two the False Discovery Rate (FDR) and the Family Wise Error Rate (FWER). In plain English, the FDR is proportion of your significant results that are false. If you control your FDR at 0.05 (or 5%), and you get 20 significant results, on average only 1 of these should be a false positive. The FWER is the probability of at least one test being a false positive. This tends to be much stricter as the number of tests grows. We can write both as follows, where $V$ is the number of tests that are statistically significant but are false positives and $R$ are the number of tests that are statistically significant. $$\text{False Discovery Rate} = \text{FDR} = \text{E}[\frac{V}{R}]$$ $$\text{Family-wise Error Rate} = \text{FWER} = \text{Pr}(V \geq 1)$$ ## Corrections in GrowthBook ### Statistical methods Growthbook allows you to choose no multiple testing correction, a correction that controls the Family-wise Error Rate (Holm-Bonferroni), and a correction that controls the False Discovery Rate (Benjamini-Hochberg). Which you choose depends on your tolerance for false positives, the number of tests you are running, and your comfort with each procedure. General guidance would be that if your analysis is more exploratory, controlling the FDR and having a slightly higher false positive rate may be better, while controlling the FWER will give you more guarantees that your results are reliable. However, with enough tests, controlling the FWER may completely undermine your test power. You can set this for your organization in the Organization → Settings page, as seen below: ![Org-level Multiple Testing Correction Settings](/images/statistics/pvals-adjustment-org.png) Once you make a selection, adjustments will happen on-the-fly in experiment results and reports. For now, these adjusted p-values are only available in the UI or in downloadable CSVs and not exportable via the API. :::tip Adjusted p-values do not have directly analogous confidence intervals The corrections we implement do not have directly analogous confidence intervals. Nonetheless, we do construct confidence intervals in an ad-hoc manner after adjusting p-values. We take the adjusted p-values and we back out what the experiment effect standard deviation would need to be to produce that adjusted p-value. We then construct CIs using this adjusted standard deviation. In this way, adjusted p-values below your p-value threshold should always coincide with confidence intervals that do not cross zero. Some adjusted p-values are 1, in which case the ad-hoc confidence interval constructed in this way is undefined. Unadjusted confidence intervals are available in the metric tooltip. ::: #### Holm-Bonferroni (controlling FWER) To control the FWER, we implement the [Holm-Bonferroni method](https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method). This method is an adaptation of the well known Bonferroni method, which simply multiplies p-values by the number of tests in the family. The Holm-Bonferroni does just as well as the Bonferroni method to control the FWER, but it is less conservative. The main trade-off is that the implementation is slightly more complex, and one cannot adjust the confidence intervals in a meaningful way. There are other approaches that are even less conservative, but they require making assumptions about the dependence between tests to guarantee FWER control. #### Benjamini-Hochberg (controlling FDR) To control the FDR, we implement the [Benjamini-Hochberg procedure](https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini%E2%80%93Hochberg_procedure). This method does assume that the tests are independent or positively correlated. There are methods that make fewer assumptions (such as the Benjamini-Yekutieli method), but they can be even more conservative than Bonferroni corrections. Therefore, in order to provide a reasonably powered approach that controls the FDR in some conditions, and to select an approach that has widespread adoption, we implemented the Benjamini-Hochberg procedure. ### Defining a family of tests Controlling for multiple comparisons across all of the tests one can run in GrowthBook, even within the context of one experiment, is impractical and can be prohibitively costly in terms of statistical power. Instead, GrowthBook corrects across all Goal metrics, variations, (and dimension values, if available) in a particular result view. :::tip Adjusted p-values are corrected using all Goal metrics We correct using all p-values for all **Goal** metrics; this excludes Secondary and Guardrail metrics. ::: #### On overall results On the _experiment results overall page_, we collect all tests across Goal metrics and variations to be part of a family, and correct within those values. The following image circles the p-values that we collect and adjust for on the overall results page for an experiment. ![Demonstrating Correction in Results](/images/statistics/pvals-adjusted.png) #### Within dimensional slices If you look within dimensional slices, to account for the additional increase in tests from dimensional splits, we define the family of tests as all tests across all dimension groups, Goal metrics, and variations. This means for a 2-armed test, with 4 goal metrics, and a dimension with 3 levels, we correct across a total of 24 tests. These dimension results are corrected independently from the overall results. --- ## GrowthBook Statistics # GrowthBook's Statistics GrowthBook provides both Bayesian and frequentist approaches to experiment analysis. We default to Bayesian statistics because they provide a more intuitive framework for decision making for most customers, but we provide ample tools to select between both approaches based on your experimentation needs. You can choose between the two statistics engines at the Organization or Project level. ## Bayesian Statistics Bayesian methods provides a few key advantages over frequentist approaches. First, they provide more intuitive results. Instead of p-values and confidence intervals, you get probabilities and distributions of likely outcomes. These values allow you to make statements like _"there’s a 95% chance this new button is better and a 5% chance it’s worse"_ while there is no direct analog in a frequentist framework. Second, Bayesian methods allow you to write down your prior knowledge about experiment effects to ensure that you do not over-interpret small sample sizes and to benefit from old knowledge to reduce uncertainty in new experiments. Third, Bayesian results are still valid even if you stop an experiment early. While they can suffer from the same "peeking" problems as frequentist statistics, at least the main probabilities and statistical results that you see are not invalidated by stopping early. However, this is something of a difference without a distinction, as the decision to stop an experiment early can still result in inflated false positive rates. However, they can require a bit more fine-tuning to get right; however, in GrowthBook both engines are very similar under the hood so picking one can come largely down to personal preference and familiarity. In fact, tools like CUPED are available for both engines. ### Priors and Posteriors At GrowthBook, we default to an improper, uninformative prior. This means that we do not use prior information to impact your experiment results by default. We do this to ensure that people who want to use the Bayesian engine to the fullest are able to enable proper priors and reap their benefits, but without automatically affecting results when experimenters first begin experimenting with GrowthBook. A prior works by providing additional information to your analysis for what kinds of results are likely based on past evidence. Our statistics engine will combine it with the actual results from your experiment to come up with our final distribution (our "posterior"). This represents the most likely outcomes of your experiment combining prior knowledge and the experiment data. Priors can be very helpful in reducing uncertainty in small sample sizes and in ensuring that you do not over-interpret results that are unlikely to be reliable. You can easily turn on proper priors by visiting the organization settings, going to the Bayesian engine settings, and turning on the "Proper Prior". By default, we use a Normal distribution with mean 0 and standard deviation 0.3. This prior implies that about 68% of effects are between -30% and 30%, 95% of effects are between -60% and 60%, and the average effect is 0%. In effect, it will shrink positive and negative results towards 0, but eventually will be overcome as more data is collected. The choice of 0 and 0.3 corresponds roughly to the distribution of effects that we have actually observed on GrowthBook and is both: 1. Weak enough to not shrink experiments with large sample sizes 2. Strong enough to ensure that experiments with small sample sizes are not over-interpreted You can read more about how we use the prior and your experiment data to produce experiment results in our [detailed documentation](/statistics/details). ### Inferential Statistics GrowthBook uses fast estimation techniques to quickly generate inferential statistics at scale for every metric in an experiment - Chance to Win and Relative Uplift (along with Absolute Change and Scaled Impact). **Chance to Win** is straight forward. It is simply the probability that a variation is better. You typically want to wait until this reaches 95% (or 5% if it's worse). **Relative Uplift** is similar to a frequentist Confidence Interval. Instead of showing a fixed 95% interval, we show the full probability distribution using a violin plot: ![Violin plot of a metrics change](/images/violin-plot.png) We have found this tends to lead to more accurate interpretations. For example, instead of just reading the above as _"it’s 17% better"_, people tend to factor in the error bars (_"it’s about 17% better, but there’s a lot of uncertainty still"_). GrowthBook gives the human decision maker everything they need to weigh the results against external factors to determine when to stop an experiment and which variation to declare the winner. ## Frequentist Statistics Frequentist statistics are are familiar to many practitioners, power much of our statistics engine, and have certain advantages. Their widespread adoption has spurred important developments in variance reduction, heterogeneous treatment effect detection, and indeed corrections to peeking issues (e.g. sequential testing) that make frequentist statistics less problematic and, at times, more valuable. The current frequentist engine computes two-sample t-tests for relative percent change; you can reduce variance (via [CUPED](/statistics/cuped)) and you can enable [sequential testing](/statistics/sequential) to mitigate concerns with peeking. You can read more in our [detailed documentation](/statistics/details). ## Data Quality Checks In addition, GrowthBook performs automatic data quality checks to ensure the statistical inferences are valid and ready for interpretation. We currently run a number of checks and plan to add even more in the future. 1. **Sample Ratio Mismatch** (SRM) detects when the traffic split doesn't match what you are expecting (e.g. a 48/52 split when you expect it to be 50/50) 2. **Multiple Exposures** which alerts you if too many users were exposed to multiple variations of a single experiment (e.g. someone saw both A and B) 3. **Guardrail Metrics** help ensure an experiment isn't inadvertently hurting core metrics like error rate or page load time 4. **Minimum Data Thresholds** so you aren't drawing conclusions too early (e.g. when it's 5 vs 2 conversions) 5. **Variation Id Mismatch** which can detect missing or improperly-tagged rows in your data warehouse 6. **Suspicious Uplift Detection** which alerts you when a metric changes by too much in a single experiment, indicating a likely bug Many of these checks are customizable at a per-metric level. So you can, for example, have stricter quality checks for revenue than you have for less important metrics. ## Dimensional Analysis There is often a desire to drill down into results to see how segments or dimensions of your users were affected by an A/B test variation. This is especially useful for finding bugs (e.g. if Safari is down, but the other browsers are up) and for identifying ideas for follow-up experiments (e.g. "European countries seem to be responding really well to this test, let's try a dedicated variation for them"). However, too much slicing and dicing of data can lead to what is known as the Multiple Testing Problem. If you look at the data in enough ways, one of them will look significant just by random chance. GrowthBook only provides multiple testing corrections for the frequentist engine, but we have a few guardrails at the metric level to ensure that results are only shown when there's at least enough data to reliably learn about a specific dimension. In addition, we apply automatic grouping to very high-cardinality dimensions. In the country example, only the top 20 countries will be shown individually. The rest will be lumped together into an `(other)` category. We have found this to be a good trade-off between false positives and false negatives. ## Conclusion GrowthBook utilizes a combination of Bayesian and frequentist statistics, fast estimation techniques, and data quality checks to robustly analyze A/B tests at scale and provide intuitive results to decision makers. The implementation is fully open source under an MIT license and available on GitHub. --- ## Post-stratification Technical Details import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Technical post-stratification details Here we document the technical details behind GrowthBook regression adjustment and post-stratification. This approach permits estimation of absolute and relative effects, and unadjusted and CUPED inference, for binomial, count, and ratio metrics. Throughout the document we describe CUPED estimation for ratio metrics, and then discuss simpler cases (e.g., non-adjusted estimates, count metrics). We assume data are available in each cell, i.e., either the number of cells is not too big, or we have already aggregated some cells together. For each case there are 4 steps. 1. In [Regression](#regression) we describe how to construct regression estimates of the treatment effect and control mean for each cell (i.e. dimension level). 2. In [Cell Moments](#cell-moments) we describe how to construct cell-specific estimates of absolute treatment effects and control means using cell-specific summary statistics. 3. In [Combining Cell Estimates](#combining-cell-estimates) we describe how to combine estimates across cells to estimate population effects and population control means. 4. Finally, in [Delta Method](#delta-method) we transform the combined estimates into estimates of lift, ratio parameters, etc. ## Regression Below we describe regression models for each cell, or dimension level. The regression output will be used in [the next section](#cell-moments) to construct the joint sampling distribution of effect estimates and control means within a stratification cell. We do this for ratio metrics, and discuss along the way the simpler case of count metrics. - Define $m_{i1}$ ($d_{i1}$) as the numerator (denominator) outcome for the $i^{\text{th}}$ user, $i=1,2,..., N$ - Define $x_{im}$ ($x_{id}$) as the pre-exposure numerator (denominator) variable for the $i^{\text{th}}$ user. - Define $w_{i}$ as the binary treatment assignment for the $i^{\text{th}}$ user. - Define the covariate vector $\textbf{x}_{i} = \left(1, w_{i}, x_{im}, x_{id}\right)$. - Define the $N \times 4$ design matrix $\tilde{\textbf{X}}$ whose $i^{\text{th}}$ row equals $\textbf{x}_{i}$. - Define the $2N \times 8$ design matrix $\textbf{X} = \textbf{I}_{2}\otimes \tilde{\textbf{X}}$. - Define the $2N$ length vector $\boldsymbol{Y} = \left\{m_{11}, d_{11},m_{21}, d_{21},..., m_{N1}, d_{N1}\right\}$. - Define the regression coefficients as $\boldsymbol{\gamma}$. Our model is of the form $$\textbf{Y} = \textbf{X}\boldsymbol{\gamma} + \textbf{E}.$$ The least squares solution for the $8 \times 1$ vector of regression coefficients $\boldsymbol{\gamma}$ is $$ \hat{\boldsymbol{\gamma}} = \left(\textbf{X}^{\top}\textbf{X} \right)^{-1}\textbf{X}^{\top}\textbf{Y}. $$ - Define $\tilde{\boldsymbol{E}}$ as the $N \times 2$ matrix of residuals, whose first column corresponds to the residuals for the numerator and the second column is the residuals for the denominator. - Define the $2\times 2$ covariance of $\textbf{E}$ as $\boldsymbol{\Psi}$. The covariance of $\hat{\boldsymbol{\gamma}}$ is $$ \boldsymbol{\Sigma}_{\boldsymbol{\gamma}} = \text{Cov}\left(\hat{\boldsymbol{\gamma}} \right) = \boldsymbol{\Psi}\otimes \left(\textbf{X}^{\top}\textbf{X} \right)^{-1}. $$ By Lyapunov's central limit theorem, $$ \hat{\boldsymbol{\gamma}} \stackrel{}{\sim} \mathcal{N}\left(\boldsymbol{\gamma}, \boldsymbol{\Sigma}_{\boldsymbol{\gamma}} \right). $$ ## Cell moments In this section we describe how to use the regression output from [the previous section](#regression) to construct the joint sampling distribution of effect estimates and control means within a stratification cell. In the $k^{\text{th}}$ cell, our inferential focus is the vector $\boldsymbol{\alpha}_{k}$, which has four elements: - numerator absolute effect estimate for the $k^{\text{th}}$ cell - numerator control mean for the $k^{\text{th}}$ cell - denominator absolute effect estimate for the $k^{\text{th}}$ cell - denominator control mean for the $k^{\text{th}}$ cell Now that we have our summary statistics in the form of a multivariate CLT, we linearly transform them to create our estimates of numerator and denominator effects and control means. - Define $\bar{x}_{m}$ ($\bar{x}_{d}$) as the sample mean pre-exposure numerator (denominator) variable. - Define $\mu_{xm}$ and $\mu_{xd}$ as their population counterparts. - Define the $4\times 8$ contrast matrix $\textbf{A}_{k, reg}$ where $$ \begin{align} \textbf{A}_{k, reg} = \begin{pmatrix} 1 & 0 & \bar{x}_{m} & \bar{x}_{d} & 0 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 1 & 0 & \bar{x}_{m} & \bar{x}_{d}\\ 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \end{pmatrix}. \end{align} $$ We estimate $\boldsymbol{\alpha}_{k}$ with $\boldsymbol{\hat{\alpha}}_{k} = \textbf{A}_{k, reg}\hat{\boldsymbol{\gamma}}_{k}$. We now calculate the covariance of $\boldsymbol{\hat{\alpha}}_{k}$, denoted as $\boldsymbol{\Sigma}_{k}$. Many readers may want to skip to the next section, [Combining cell estimates](#combining-cell-estimates), where we describe how to combine estimates across cells to estimate population absolute effects and control means. One subtlety is that $\textbf{A}_{k, reg}$ has random components which must be accounted for. For inference within a cell, we condition upon the sample size for that cell. We deal with the assignment randomness in the next section. Technically, each of the covariances and expectations below are conditional upon $n_{k}$, but we suppress this notation for clarity. Below we describe how to calculate row means and covariances between individual rows of $\textbf{A}_{k, reg}$. The first moment of $\textbf{A}_{k, reg}$ is $$ \begin{align} E\left[\textbf{A}_{k, reg}\right] &= E\left[ \begin{pmatrix} 1 & 0 & \bar{x}_{m} & \bar{x}_{d} & 0 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 1 & 0 & \bar{x}_{m} & \bar{x}_{d}\\ 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \end{pmatrix} \right] \\&= \begin{pmatrix} 1 & 0 & \mu_{xm} & \mu_{xd} & 0 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 1 & 0 & \mu_{xm} & \mu_{xd}\\ 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \end{pmatrix}. \end{align} $$ We also need the covariance between individual rows of $\textbf{A}_{k, reg}$. Note that there is nothing random in the second and fourth rows of $\textbf{A}_{k, reg}$, so the covariance of any vectors with these terms is 0. There are only 4 cases we need to consider, and we start with the covariance of the first row of $\textbf{A}_{k, reg}$ with itself. $$ \begin{align*} Cov\left(\textbf{A}_{k, reg}[1, ], \textbf{A}_{k, reg}[1, ]\right) &= E\left[ \begin{pmatrix} 1 & 0 & \bar{x}_{m} & \bar{x}_{d} & 0 & 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} 1\\ 0\\ \bar{x}_{m} \\ \bar{x}_{d} \\ 0\\ 0\\ 0 \\ 0\\ \end{pmatrix} \right] \\&- E\left[ \begin{pmatrix} 1 & 0 & \bar{x}_{m} & \bar{x}_{d} & 0 & 0 & 0 & 0 \end{pmatrix} \right] E\left[ \begin{pmatrix} 1\\ 0\\ \bar{x}_{m} \\ \bar{x}_{d} \\ 0\\ 0\\ 0 \\ 0\\ \end{pmatrix} \right] \\&= \begin{pmatrix} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & \sigma_{xm}^{2}/n & \sigma_{xmd}/n & 0 & 0 & 0 & 0\\ 0 & 0 & \sigma_{xmd}/n & \sigma_{xd}^{2}/n & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \end{pmatrix}. \end{align*} $$ Using a similar argument for the (3, 3) case: $$ \begin{align*} Cov\left(\textbf{A}_{k, reg}[3, ], \textbf{A}_{k, reg}[3, ]\right) &= \\&= \begin{pmatrix} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & \sigma_{xm}^{2}/n & \sigma_{xmd}/n\\ 0 & 0 &0 & 0 & 0 & 0 & \sigma_{xmd}/n & \sigma_{xd}^{2}/n \end{pmatrix}. \end{align*} $$ For the (1, 3) case: $$ \begin{align*} Cov\left(\textbf{A}_{k, reg}[1, ], \textbf{A}_{k, reg}[3, ]\right) &= E\left[ \begin{pmatrix} 1 & 0 & \bar{x}_{m} & \bar{x}_{d} & 0 & 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} 0\\ 0\\ 0\\ 0\\ 1 \\ 0\\ \bar{x}_{m} \\ \bar{x}_{d} \end{pmatrix} \right] \\&- E\left[ \begin{pmatrix} 1 & 0 & \bar{x}_{m} & \bar{x}_{d} & 0 & 0 & 0 & 0 \end{pmatrix} \right] E\left[ \begin{pmatrix} 0\\ 0\\ 0\\ 0\\ 1 \\ 0\\ \bar{x}_{m} \\ \bar{x}_{d} \end{pmatrix} \right] \\&= \begin{pmatrix} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & \sigma_{xm}^{2}/n & \sigma_{xmd}/n & 0 & 0 & 0 & 0\\ 0 & 0 & \sigma_{xmd}/n & \sigma_{xd}^{2}/n & 0 & 0 & 0 & 0 \end{pmatrix} \end{align*} $$ For the (3, 1) case: $$ \begin{align*} Cov\left(\textbf{A}_{k, reg}[3, ], \textbf{A}_{k, reg}[1, ]\right) = Cov\left(\textbf{A}_{k, reg}[1, ], \textbf{A}_{k, reg}[3, ]\right)' \end{align*} $$ Define $\boldsymbol{\mu}_{k, reg}$ as the mean of $\textbf{A}_{k, reg}$. $$ \begin{align*} \text{Cov}\left(\hat{\alpha}_{k}\right) &= \text{Cov}\left(\textbf{A}_{k, reg}\hat{\gamma}_{k}\right) \\&= E\left[\text{Cov}\left(\textbf{A}_{k, reg}\hat{\gamma}_{k}\right)|\textbf{A}_{k, reg}\right] + \text{Cov}\left[E\left(\textbf{A}_{k, reg}\hat{\gamma}_{k}\right)|\textbf{A}_{k, reg}\right] \\&= E\left[\textbf{A}_{k, reg}\text{Cov}\left(\hat{\gamma}_{k}\right)\textbf{A}_{k, reg}^{\top}\right] + \text{Cov}\left[\textbf{A}_{k, reg}\gamma_{k}\right] \end{align*} $$ The first term has $(i,j)^{\text{th}}$ element equal to $$ \begin{align*} E\left[\textbf{A}_{k, reg}\text{Cov}\left(\hat{\boldsymbol{\gamma}}_{k}\right)\textbf{A}_{k, reg}^{\top}\right][i,j] &= E\left[\textbf{A}_{k, reg}[i, ]\text{Cov}\left(\hat{\boldsymbol{\gamma}}_{k}\right)\textbf{A}_{k, reg}[j, ]^{\top}\right] \\&= E\left[ \text{trace}\left(\textbf{A}_{k, reg}[i, ]\text{Cov}\left(\hat{\boldsymbol{\gamma}}_{k}\right)\textbf{A}_{k, reg}[j, ]^{\top}\right) \right] \\&= E\left[ \text{trace}\left(\text{Cov}\left(\hat{\boldsymbol{\gamma}}_{k}\right)\textbf{A}_{k, reg}[j, ]^{\top}\textbf{A}_{k, reg}[i, ]\right) \right] \\&= \text{trace}\left(\text{Cov}\left(\hat{\boldsymbol{\gamma}}_{k}\right) E\left[ \textbf{A}_{k, reg}[j, ]^{\top}\textbf{A}_{k, reg}[i, ]\right) \right] \end{align*} $$ A similar argument exists for the second term. Therefore, $$ \begin{align*} \boldsymbol{\Sigma}_{k}[i, j] &= \text{Cov}\left(\boldsymbol{\hat{\alpha}}_{k}\right)[i, j] &= \text{trace}\left(\text{Cov}\left(\hat{\boldsymbol{\gamma}}_{k}\right) E\left[ \textbf{A}_{k, reg}[j, ]\textbf{A}_{k, reg}[i, ]^{\top}\right) \right] +\text{trace}\left( \boldsymbol{\gamma}_{k} \boldsymbol{\gamma}_{k}^{\top} \text{Cov}\left[ \textbf{A}_{k, reg}[j, ]\textbf{A}_{k, reg}[i, ]^{\top} \right] \right) \end{align*} $$ In practice, we substitute $\boldsymbol{\gamma}_{k}$ for $\hat{\boldsymbol{\gamma}}_{k}$. ## Combining cell estimates At a high level, for each cell we now have estimates of population means (e.g., the control mean and the absolute effect), and uncertainty about those estimates. In this section we describe how to combine these estimates across cells to estimate population absolute effects and control means. This algorithm can be used for count or ratio metrics, unadjusted or adjusted (e.g., CUPED), and absolute or relative inference. Define the population (sample) proportion for the $k^{\text{th}}$ strata cell as $\nu_{k}$ ($\hat{\nu}_{k})$. Under stratified sampling, the $\nu_{k}$ are deterministic, and we could define $\hat{\boldsymbol{\alpha}} = \sum_{k=1}^{K}\nu_{k}\hat{\boldsymbol{\alpha}}_{k}$ and $\hat{\boldsymbol{\Sigma}} = \sum_{k=1}^{K}\nu_{k}^{2}n_{k}^{-1}\hat{\boldsymbol{\Sigma}}_{k}$. However, we do not conduct stratified sampling in GrowthBook. Under simple random sampling the $\hat{\nu}_{k}$ are multinomial random variables, and we could define $\hat{\boldsymbol{\alpha}} = \sum_{k=1}^{K}\hat{\nu}_{k}\hat{\boldsymbol{\alpha}}_{k}$. Define the $4 \times K$ matrix $\boldsymbol{\alpha}_{M}$ as the matrix whose $k^{\text{th}}$ row is $\boldsymbol{\alpha}_{k}$. Our point estimate is the expected value of $\sum_{k=1}^{K}\hat{\nu_{k}}\hat{\boldsymbol{\alpha}}_{k}$, which is $$ \begin{align*} E\left(\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}} \right) &= E_{\hat{\boldsymbol{\nu}}}\left(E_{\hat{\boldsymbol{\alpha}}}\left(\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}}|\hat{\boldsymbol{\nu}} \right)\right) \\&= \boldsymbol{\alpha}_{M}E_{\hat{\boldsymbol{\nu}}}\left( \hat{\boldsymbol{\nu}} \right) \\&= \boldsymbol{\alpha}_{M}\boldsymbol{\nu}. \end{align*} $$ Below we derive its covariance. Define the collection of $\hat{\nu}_{k}$ as $\hat{\boldsymbol{\nu}}$. The naive covariance is $$\hat{\boldsymbol{\Sigma}} = \sum_{k=1}^{K}\hat{\nu}_{k}^{2}n_{k}^{-1}\hat{\boldsymbol{\Sigma}}_{k}$$. Alternatively, we can use Equation 15 in ([Xie and Aurriset 2016](https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf)) to define $$ \hat{\boldsymbol{\Sigma}} = n^{-1}\sum*{k=1}^{K}\left(\boldsymbol{\nu}[k] + \frac{1-\boldsymbol{\nu}[k]}{n} \right)\boldsymbol{\Sigma}*{k} $$ Both approaches assume the population cell proportions $\nu_{k}$ are known. For GrowthBook experiments, the $\hat{\nu}_{k}$ are random variables, and this assumption is not met. There is dependence between the $n_{k}$ (or equivalently, between the $\hat{\boldsymbol{\nu}}_{k}$) that is not accounted for when estimating the variance. We show below in the Section [Derivation of conditional covariance](#derivation-of-conditional-covariance) that $\text{Cov}\left(\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}} \right)$ is: $$ \begin{align} \text{Cov}\left(\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}} \right) &= \boldsymbol{\alpha}_{M}\text{Cov}\left(\hat{\boldsymbol{\nu}}\right)\boldsymbol{\alpha}_{M}^{\top} + n^{-1}\sum_{k=1}^{K} \boldsymbol{\nu}[k]\boldsymbol{\Sigma}_{k}. \end{align} $$ Note that $\hat{\boldsymbol{\nu}}$ is a multinomial random variable divided by $n$, so its $K\times K$ covariance matrix has diagonal element equal to $\nu_{k} / n^{2}$ and off-diagonal element $(i, j)^{\text{th}}$ element equal to $-\nu_{i} \nu_{j} / n^{2}$. ## Delta method To recapitulate, we now have an estimate of the joint sampling distribution of the vector $\boldsymbol{\alpha}$, which has four elements: - numerator absolute effect estimate - numerator control mean - denominator absolute effect estimate - denominator control mean. To estimate lift (relative effects), we use the delta method. ### Delta method for ratio metrics By the central limit theorem $$ \begin{equation} \hat{\boldsymbol{\alpha}} =\begin{pmatrix} \hat{\boldsymbol{\alpha}}_{1}\\ \hat{\boldsymbol{\alpha}}_{2} \\ \hat{\boldsymbol{\alpha}}_{3} \\ \hat{\boldsymbol{\alpha}}_{4} \end{pmatrix}\stackrel{}{\sim}\mathcal{N}\left(\boldsymbol{\alpha}=\begin{pmatrix} \boldsymbol{\alpha}_{1}\\ \boldsymbol{\alpha}_{2} \\ \boldsymbol{\alpha}_{3} \\ \boldsymbol{\alpha}_{4} \end{pmatrix},\boldsymbol{\Sigma}\right) \end{equation} $$ Define $$g_{abs}(\boldsymbol{\alpha}) = \frac{\boldsymbol{\alpha}[1] + \boldsymbol{\alpha}[2]}{\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]} - \frac{\boldsymbol{\alpha}[1]}{\boldsymbol{\alpha}[3]}$$. Define $$ \begin{align*} g_{rel}(\boldsymbol{\alpha}) &= \frac{ \frac{\boldsymbol{\alpha}[1] + \boldsymbol{\alpha}[2]}{\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]} - \frac{\boldsymbol{\alpha}[1]}{\boldsymbol{\alpha}[3]}}{\boldsymbol{\alpha}[1] / \boldsymbol{\alpha}[3]} \\&= \frac{\frac{\boldsymbol{\alpha}[1] + \boldsymbol{\alpha}[2]}{\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]}}{\boldsymbol{\alpha}[1] / \boldsymbol{\alpha}[3]} - 1 \\&= \frac{\boldsymbol{\alpha}[3]\left(\boldsymbol{\alpha}[1] + \boldsymbol{\alpha}[2]\right)}{\boldsymbol{\alpha}[1]\left(\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]\right)}- 1 \\&=\frac{g_{rel, N}}{g_{rel, D}} - 1. \end{align*} $$ Define $g \in \left\{g_{abs}, g_{rel} \right\}$. Define the vector of partials of length $4$ $\boldsymbol{\nabla} = \frac{\partial g}{\partial \boldsymbol{\alpha}}$. If $g = g_{abs}$ then set $\boldsymbol{\nabla}$ equal to $\boldsymbol{\nabla}_{abs}$, where $$ \begin{align*} \boldsymbol{\nabla}_{abs}[1] &= \frac{1}{\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]}- \frac{1}{\boldsymbol{\alpha}[3]}\\ \boldsymbol{\nabla}_{abs}[2] &= \frac{1}{\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]}\\ \boldsymbol{\nabla}_{abs}[3] &= \frac{-\left(\left[\boldsymbol{\alpha}[1] + \boldsymbol{\alpha}[2]\right]\right)}{ \left(\left[\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]\right]\right)^{2} } + \frac{\boldsymbol{\alpha}[1]}{\boldsymbol{\alpha}[3]^{2}}\\ \boldsymbol{\nabla}_{abs}[4] &= \frac{-\left(\left[\boldsymbol{\alpha}[1] + \boldsymbol{\alpha}[2]\right]\right)}{ \left(\left[\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]\right]\right)^{2} } \end{align*} $$ If $g = g_{rel}$ then define $\boldsymbol{\nabla}$ equal to $\boldsymbol{\nabla}_{rel}$, where $$ \begin{align*} \boldsymbol{\nabla}_{rel}[1] &= \frac{\boldsymbol{\alpha}[3]g_{rel, D} - (\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4])g_{rel, N}} {g_{rel, D}^{2}}\\ \boldsymbol{\nabla}_{rel}[2] &= \frac{\boldsymbol{\alpha}[3]}{g_{rel, D}}\\ \boldsymbol{\nabla}_{rel}[3] &= \frac{(\boldsymbol{\alpha}[1] + \boldsymbol{\alpha}[2])g_{rel, D} - \boldsymbol{\alpha}[1] g_{rel, N}} {g_{rel, D}^{2}}\\ \boldsymbol{\nabla}_{rel}[4] &= \frac{-\boldsymbol{\alpha}[3]\left(\boldsymbol{\alpha}[1] + \boldsymbol{\alpha}[2]\right)}{ \boldsymbol{\alpha}[1]\left(\boldsymbol{\alpha}[3] + \boldsymbol{\alpha}[4]\right)^{2} } \end{align*} $$ By the delta method, $$\hat{\Delta}_{r} = g(\hat{\boldsymbol{\alpha}}) \stackrel{}{\sim}\mathcal{N}\left(\Delta_{r} = g\left(\alpha\right), \boldsymbol{\nabla}^{\top}\boldsymbol{\Sigma}\boldsymbol{\nabla}\right)$$. In summary, the steps for the algorithm are: 1. Compute the point estimate $\hat{\Delta} = g(\hat{\boldsymbol{\alpha}})$. 2. Compute the estimated variance $\hat{v} = \boldsymbol{\nabla}^{\top}\boldsymbol{\Sigma}\boldsymbol{\nabla}$. 3. Return $(\hat{\Delta}, \hat{v})$. ### Delta method for count metrics Define $\hat{\boldsymbol{\alpha}}$ as the $2\times 1$ vector with the control sample mean and the numerator effect estimate. Define $\hat{\boldsymbol{\Sigma}}$ as the $2 \times 2$ covariance of $\hat{\boldsymbol{\alpha}}$. By the central limit theorem $$ \begin{equation} \hat{\boldsymbol{\alpha}} =\begin{pmatrix} \hat{\boldsymbol{\alpha}}_{1}\\ \hat{\boldsymbol{\alpha}}_{2} \end{pmatrix}\stackrel{}{\sim}\mathcal{N}\left(\boldsymbol{\alpha}=\begin{pmatrix} \boldsymbol{\alpha}_{1}\\ \boldsymbol{\alpha}_{2} \end{pmatrix},\boldsymbol{\Sigma}\right) \end{equation} $$ Define $$g_{abs}(\boldsymbol{\alpha}) = \boldsymbol{\alpha}[2]$$. Define $$g_{rel}(\boldsymbol{\alpha}) = \frac{\boldsymbol{\alpha}[2]}{\boldsymbol{\alpha}[1]}.$$ Define $g \in \left\{g_{abs}, g_{rel} \right\}$. Define the vector of partials of length $2$ $\boldsymbol{\nabla} = \frac{\partial g}{\partial \boldsymbol{\alpha}}$. If $g = g_{abs}$ then set $\boldsymbol{\nabla}$ equal to $\boldsymbol{\nabla}_{abs}$, where $$ \begin{align*} \boldsymbol{\nabla}_{abs}[1] &= 0\\ \boldsymbol{\nabla}_{abs}[2] &= 1. \end{align*} $$ If $g = g_{rel}$ then define $\boldsymbol{\nabla}$ equal to $\boldsymbol{\nabla}_{rel}$, where $$ \begin{align*} \boldsymbol{\nabla}_{rel}[1] &= \frac{-\boldsymbol{\alpha}[2]} {\boldsymbol{\alpha}[1]^{2}}\\ \boldsymbol{\nabla}_{rel}[2] &= \frac{1}{\boldsymbol{\alpha}[1]}\\ \end{align*} $$ By the delta method, $$\hat{\Delta} = g(\hat{\boldsymbol{\alpha}}) \stackrel{}{\sim}\mathcal{N}\left(\Delta_{r} = g\left(\alpha\right), \boldsymbol{\nabla}^{\top}\boldsymbol{\Sigma}\boldsymbol{\nabla}\right)$$. In summary, the steps for the algorithm are: 1. Compute the point estimate $\hat{\Delta} = g(\hat{\boldsymbol{\alpha}})$. 2. Compute the estimated variance $\hat{v} = \boldsymbol{\nabla}^{\top}\boldsymbol{\Sigma}\boldsymbol{\nabla}$. 3. Return $(\hat{\Delta}, \hat{v})$. ## Appendix ### Derivation of conditional covariance In this section we derive the covariance of $\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}}$. We derive the covariance using results from linear models. Recall that if $\textbf{A}$ is a matrix and $\textbf{Z}$ is a random vector with mean $\boldsymbol{\mu}$ and covariance $\boldsymbol{\Psi}$ then $E\left(\textbf{Z}^{\top} \textbf{A}\textbf{Z}\right) = \boldsymbol{\alpha}^{\top} \textbf{A}\boldsymbol{\alpha} + \text{tr}\left( \textbf{A}\boldsymbol{\Psi}\right).$ Define $\boldsymbol{\alpha}_{l}$ as the $l^{\text{th}}$ row of $\boldsymbol{\alpha}_{M}$, and analogously define $\hat{\boldsymbol{\alpha}}_{m}$. Below we derive $\text{Cov}\left(\hat{\boldsymbol{\alpha}}_{M}|\hat{\boldsymbol{\nu}}\right)$. Define $\boldsymbol{\alpha}_{l}$ as the $l^{\text{th}}$ row of $\boldsymbol{\alpha}_{M}$, and analogously define $\hat{\boldsymbol{\alpha}}_{m}$. First we need the following result: $E\left(\hat{\boldsymbol{\alpha}}_{l}\hat{\boldsymbol{\alpha}}_{m}^{\top}|\hat{\boldsymbol{\nu}}\right)$ $$ \begin{align*} &= \begin{bmatrix} E\left(\hat{\boldsymbol{\alpha}}_{l}[1]\hat{\boldsymbol{\alpha}}_{m}[1]|\hat{\boldsymbol{\nu}}\right) & E\left(\hat{\boldsymbol{\alpha}}_{l}[1]\hat{\boldsymbol{\alpha}}_{m}[2]|\hat{\boldsymbol{\nu}}\right) & ... & E\left(\hat{\boldsymbol{\alpha}}_{l}[1]\hat{\boldsymbol{\alpha}}_{m}[i, K]|\hat{\boldsymbol{\nu}}\right) \\ E\left(\hat{\boldsymbol{\alpha}}_{l}[2]\hat{\boldsymbol{\alpha}}_{m}[1]|\hat{\boldsymbol{\nu}}\right) & E\left(\hat{\boldsymbol{\alpha}}_{l}[2]\hat{\boldsymbol{\alpha}}_{m}[2]|\hat{\boldsymbol{\nu}}\right) & ... & E\left(\hat{\boldsymbol{\alpha}}_{l}[2]\hat{\boldsymbol{\alpha}}_{m}[i, K]|\hat{\boldsymbol{\nu}}\right) \\ \vdots & \vdots & \vdots & \vdots \\ E\left(\hat{\boldsymbol{\alpha}}_{l}[K]\hat{\boldsymbol{\alpha}}_{m}[1]|\hat{\boldsymbol{\nu}}\right) & E\left(\hat{\boldsymbol{\alpha}}_{l}[K]\hat{\boldsymbol{\alpha}}_{m}[2]|\hat{\boldsymbol{\nu}}\right) & ... & E\left(\hat{\boldsymbol{\alpha}}_{l}[K]\hat{\boldsymbol{\alpha}}_{m}[i, K]|\hat{\boldsymbol{\nu}}\right) \\ \end{bmatrix} \\&= \begin{bmatrix} E\left(\hat{\boldsymbol{\alpha}}_{l}[1]\hat{\boldsymbol{\alpha}}_{m}[1]|\hat{\boldsymbol{\nu}}\right) & E\left(\hat{\boldsymbol{\alpha}}_{l}[1]|\hat{\boldsymbol{\nu}}\right)E\left(\hat{\boldsymbol{\alpha}}_{m}[2]|\hat{\boldsymbol{\nu}}\right) & ... & E\left(\hat{\boldsymbol{\alpha}}_{l}[1]|\hat{\boldsymbol{\nu}}\right)E\left(\hat{\boldsymbol{\alpha}}_{m}[i, K]|\hat{\boldsymbol{\nu}}\right) \\ E\left(\hat{\boldsymbol{\alpha}}_{l}[2]|\hat{\boldsymbol{\nu}}\right)E\left(\hat{\boldsymbol{\alpha}}_{m}[1]|\hat{\boldsymbol{\nu}}\right) & E\left(\hat{\boldsymbol{\alpha}}_{l}[2]\hat{\boldsymbol{\alpha}}_{m}[2]|\hat{\boldsymbol{\nu}}\right) & ... & E\left(\hat{\boldsymbol{\alpha}}_{l}[2]|\hat{\boldsymbol{\nu}}\right)E\left(\hat{\boldsymbol{\alpha}}_{m}[i, K]|\hat{\boldsymbol{\nu}}\right) \\ \vdots & \vdots & \vdots & \vdots \\ E\left(\hat{\boldsymbol{\alpha}}_{l}[K]|\hat{\boldsymbol{\nu}}\right)E\left(\hat{\boldsymbol{\alpha}}_{m}[1]|\hat{\boldsymbol{\nu}}\right) & E\left(\hat{\boldsymbol{\alpha}}_{l}[1]|\hat{\boldsymbol{\nu}}\right)E\left(\hat{\boldsymbol{\alpha}}_{m}[i, K]|\hat{\boldsymbol{\nu}}\right) & ... & E\left(\hat{\boldsymbol{\alpha}}_{l}[K]|\hat{\boldsymbol{\nu}}\right)E\left(\hat{\boldsymbol{\alpha}}_{m}[i, K]|\hat{\boldsymbol{\nu}}\right) \end{bmatrix} \end{align*} $$ Therefore, $$ \begin{align*} \text{Cov}\left(\hat{\boldsymbol{\alpha}}_{l}, \hat{\boldsymbol{\alpha}}_{m}|\hat{\boldsymbol{\nu}}\right) &= E\left(\hat{\boldsymbol{\alpha}}_{l}\hat{\boldsymbol{\alpha}}_{m}^{\top}|\hat{\boldsymbol{\nu}}\right) - E\left(\hat{\boldsymbol{\alpha}}_{l}|\hat{\boldsymbol{\nu}}\right)E\left(\hat{\boldsymbol{\alpha}}_{m}^{\top}|\hat{\boldsymbol{\nu}}\right) \\&= \begin{bmatrix} \text{Cov}\left(\hat{\boldsymbol{\alpha}}_{l}[1], \hat{\boldsymbol{\alpha}}_{m}[1]|\hat{\boldsymbol{\nu}}\right) & 0 & ... & 0 \\ 0 & \text{Cov}\left(\hat{\boldsymbol{\alpha}}_{l}[1], \hat{\boldsymbol{\alpha}}_{m}[1]|\hat{\boldsymbol{\nu}}\right) & ... & 0 \\ \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & ... & \text{Cov}\left(\hat{\boldsymbol{\alpha}}_{l}[K], \hat{\boldsymbol{\alpha}}_{m}[K]|\hat{\boldsymbol{\nu}}\right) \end{bmatrix} \\&= n^{-1}\begin{bmatrix} \boldsymbol{\Sigma}_{1}[l, m] / (\hat{\boldsymbol{\nu}}[1]) & 0 & ... & 0 \\ 0 & \boldsymbol{\Sigma}_{2}[l, m] / (\hat{\boldsymbol{\nu}}[2]) & ... & 0 \\ \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & ... & \boldsymbol{\Sigma}_{K}[l, m] / (\hat{\boldsymbol{\nu}}[K]) \end{bmatrix} \end{align*} $$ To get the $(l, m)^{\text{th}}$ element of the covariance we first calculate the $(l, m)^{\text{th}}$ element of the second moment $E\left(\hat{\boldsymbol{\alpha}}_{l}\hat{\boldsymbol{\nu}}\hat{\boldsymbol{\nu}}^{\top}\hat{\boldsymbol{\alpha}}_{m}^{\top} \right)$: $$ \begin{align*} &= E_{\hat{\boldsymbol{\nu}}}\left(E_{\hat{\boldsymbol{\alpha}}}\left(\hat{\boldsymbol{\alpha}}_{l}\hat{\boldsymbol{\nu}}\hat{\boldsymbol{\nu}}^{\top}\hat{\boldsymbol{\alpha}}_{m}^{\top}|\hat{\boldsymbol{\nu}} \right)\right) \\&= E_{\hat{\boldsymbol{\nu}}} \left( \boldsymbol{\alpha}_{l}\hat{\boldsymbol{\nu}}\hat{\boldsymbol{\nu}}^{\top}\boldsymbol{\alpha}_{m}^{\top} \right) \\&+ E_{\hat{\boldsymbol{\nu}}} \left( \text{tr}\left(\hat{\boldsymbol{\nu}}\hat{\boldsymbol{\nu}}^{\top}\text{Cov}\left(\hat{\boldsymbol{\alpha}}_{l}, \hat{\boldsymbol{\alpha}}_{m}|\hat{\boldsymbol{\nu}} \right)\right)|\hat{\boldsymbol{\nu}} \right) \\&= E_{\hat{\boldsymbol{\nu}}} \left( \boldsymbol{\alpha}_{l}\hat{\boldsymbol{\nu}}\hat{\boldsymbol{\nu}}^{\top}\boldsymbol{\alpha}_{m}^{\top} \right) \\&+ E_{\hat{\boldsymbol{\nu}}} \left( \text{tr}\left(\text{Cov}\left(\hat{\boldsymbol{\alpha}}_{l}, \hat{\boldsymbol{\alpha}}_{m}|\hat{\boldsymbol{\nu}} \right)\hat{\boldsymbol{\nu}}\hat{\boldsymbol{\nu}}^{\top}\right)|\hat{\boldsymbol{\nu}} \right) \\&= E_{\hat{\boldsymbol{\nu}}} \left( \boldsymbol{\alpha}_{l}\hat{\boldsymbol{\nu}}\hat{\boldsymbol{\nu}}^{\top}\boldsymbol{\alpha}_{m}^{\top} \right) \\&+ E_{\hat{\boldsymbol{\nu}}} \left( \text{tr}\left( n^{-1}\begin{bmatrix} \boldsymbol{\Sigma}_{1}[l, m] / \hat{\boldsymbol{\nu}}[1] & 0 & ... & 0 \\ 0 & \boldsymbol{\Sigma}_{2}[l, m] / \hat{\boldsymbol{\nu}}[2] & ... & 0 \\ \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & ... & \boldsymbol{\Sigma}_{K}[l, m] / \hat{\boldsymbol{\nu}}[K] \end{bmatrix} \right) \right) \\&= \boldsymbol{\alpha}_{l}\left(\text{Cov}\left(\hat{\boldsymbol{\nu}}\right) + \boldsymbol{\nu}\boldsymbol{\nu}^{\top}\right)\boldsymbol{\alpha}_{m}^{\top} \\&+ n^{-1}\sum_{k=1}^{K} \boldsymbol{\nu}[k]\boldsymbol{\Sigma}_{k}[l, m] \end{align*} $$ where we used the fact that the trace of a product of a diagonal matrix and a matrix is the sum of the products of the diagonal elements of the diagonal matrix and the diagonal elements of the matrix. In summary, the $(l, m)^{\text{th}}$ element of $\text{Cov}\left(\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}} \right)$ is: $$ \begin{align*} \text{Cov}\left(\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}} \right)[l, m] &= \boldsymbol{\alpha}_{l}\text{Cov}\left(\hat{\boldsymbol{\nu}}\right)\boldsymbol{\alpha}_{m}^{\top} + n^{-1}\sum_{k=1}^{K} \boldsymbol{\nu}[k]\boldsymbol{\Sigma}_{k}[l, m]. \end{align*} $$ Therefore, the covariance of $\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}}$ is: $$ \begin{align*} \text{Cov}\left(\hat{\boldsymbol{\alpha}}_{M}\hat{\boldsymbol{\nu}} \right) &= \boldsymbol{\alpha}_{M}\text{Cov}\left(\hat{\boldsymbol{\nu}}\right)\boldsymbol{\alpha}_{M}^{\top} + n^{-1}\sum_{k=1}^{K} \boldsymbol{\nu}[k]\boldsymbol{\Sigma}_{k}. \end{align*} $$ --- ## Power Analysis Technical Details import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Technical power details Here we document the technical details behind GrowthBook power calculations and minimum detectable effect (MDE) calculations for both [frequentist](#frequentist-engine-details) and [Bayesian](#bayesian-engine-details) engines. ## Frequentist engine details ### Frequentist power Below we describe technical details of our implementation. First we start with the definition of power. **Power** is the probability of a statistically significant result. We use the terms below throughout. Define: 1. the false positive rate as $\alpha$ (GrowthBook default is $\alpha=0.05$). 2. the critical values $Z_{1- \alpha / 2}= \Phi^{-1}(1-\alpha/2)$ and $Z_{1-\alpha}= \Phi^{-1}(1-\alpha)$ where $\Phi^{-1}$ is the inverse CDF of the standard normal distribution. 3. the true relative treatment effect as $\Delta$, its estimate as $\hat{\Delta}$ and its estimated standard error as $\hat{\sigma}_{\Delta}$. Note that as the sample size $n$ increases, $\hat{\sigma}_{\Delta}$ decreases by a factor of $1/\sqrt{n}$. We make the following assumptions: 1. equal sample sizes across control and treatment variations. If unequal sample sizes are used in the experiment, use the smaller of the two sample sizes. This will produce conservative power estimates. 2. equal variance across control and treatment variations; 3. observations across users are independent and identically distributed; 4. all metrics have finite variance; and 5. you are running a two-sample t-test. If in practice you use CUPED, your power will be higher. For a 1-sided test, the power is $$ \begin{align} \pi = P\left(\frac{\hat{\Delta}}{\hat{\sigma}_{\Delta}} > Z_{1-\alpha}\right)=P\left(\frac{\hat{\Delta}-\Delta}{\hat{\sigma}_{\Delta}} > Z_{1-\alpha}-\frac{\Delta}{\hat{\sigma}_{\Delta}}\right)=1 - \Phi\left(Z_{1-\alpha}-\frac{\Delta}{\hat{\sigma}_{\Delta}}\right). \end{align} $$ For a 2-sided test (all GrowthBook tests are 2-sided), power is composed of the probability of a statistically significant positive result and a statistically significant negative result. Using the same algebra as in Equation 1 (except using $Z_{1-0.5\alpha}$ for the critical value), the probability of a statistically significant positive result is $$ \begin{align} \pi_{pos} &= 1 - \Phi\left(Z_{1-\alpha/2}-\frac{\Delta}{\hat{\sigma}_{\Delta}}\right). \end{align} $$ The probability of a statistically significant negative result is $$ \begin{align} \pi_{neg} &= P\left(\frac{\hat{\Delta}}{\hat{\sigma}_{\Delta}} 0 &\iff 2 > \frac{\sigma^{2}}{N} * \frac{k^{2}}{\mu_{A}^{2}} \iff N > \frac{\sigma^{2}k^{2}}{2\mu_{A}^{2}}. \end{align} $$ Similarly, the MDE returned can be negative if the denominator is negative, which is nonsensical. We return cases only where the denominator is positive, which occurs if and only if: $$ \begin{align} \left(1-\frac{\sigma^{2}}{N}\frac{k^{2}}{\mu_{A}^{2}}\right) > 0 \iff \left(1-\frac{\sigma^{2}}{N}\frac{k^{2}}{\mu_{A}^{2}}\right) > 0 \iff N > \frac{\sigma^{2}k^{2}}{\mu_{A}^{2}}. \end{align} $$ The condition in Equation 10 is stricter than the condition in Equation 9. In summary, there will be some combinations of $(\mu_{A}, \sigma_{2})$ where the MDE does not exist for a given $N$. If $\alpha=0.05$ and $\pi=0.8$, then $k\approx 2.8$. Therefore, a rule of thumb is that $N$ needs to be roughly 9 times larger than the ratio of the variance to the squared mean to return an MDE. In these cases, $N$ needs to be increased. ### Sequential testing To estimate power under sequential testing, we adjust the variance term $\hat{\sigma}_{\delta}$ to account for sequential testing, and then input this adjusted variance into our power formula. We assume that you look at the data only once, so our power estimate below is a lower bound for the actual power under sequential testing. Otherwise we would have to make assumptions about the temporal correlation of the data generating process. In sequential testing we construct confidence intervals as $$ \begin{align*} \hat{\Delta} &\pm \hat{\sigma}*\sqrt{N} * \sqrt{\frac{2(N\rho^2 + 1)}{N^2\rho^2}\log\left(\frac{\sqrt{N\rho^2 + 1}}{\alpha}\right)} \end{align*} $$ where $$\rho = \sqrt{\frac{-2\text{log}(\alpha) + \text{log}(-2 \text{log}(\alpha) + 1)}{N^*}}$$ and $N^{\star}$ is a tuning parameter. This approach relies upon asymptotic normality. For power analysis we rewrite the confidence interval as $$ \begin{align*} \hat{\Delta} &\pm \hat{\sigma}*\sqrt{N} * \sqrt{\frac{2(N\rho^2 + 1)}{N^2\rho^2}\log\left(\frac{\sqrt{N\rho^2 + 1}}{\alpha}\right)}\frac{Z_{1-\alpha/2}}{Z_{1-\alpha/2}} \\&=\hat{\Delta_{r}} \pm \tilde{\sigma}Z_{1-\alpha/2} \end{align*} $$ where $\tilde{\sigma} = \hat{\sigma}*\sqrt{N}\sqrt{\frac{2(N\rho^2 + 1)}{N^2\rho^2}\log\left(\frac{\sqrt{N\rho^2 + 1}}{\alpha}\right)}\frac{1}{Z_{1-\alpha/2}}$. We use power analysis described above, except we substitute $\tilde{\sigma}^{2}$ for $\hat{\sigma}_{\Delta}^{2}$. ## Bayesian engine details ### Bayesian power For Bayesian power analysis, we let users specify the prior distribution of the treatment effect. We then estimate Bayesian power, which is the probability that the $(1 - \alpha)$ credible interval does not contain 0. We assume a conjugate normal-normal model, as follows: $$ \begin{align*} \Delta &\stackrel{}{\sim}\mathcal{N}\left(\mu_{prior}, \sigma_{prior}^{2}\right) \\ \hat{\Delta}|\Delta &\stackrel{}{\sim}\mathcal{N}\left(\Delta, \hat{\sigma}_{\Delta}^{2}\right). \end{align*} $$ In words, the model has two parts: 1) the normal prior for the treatment effect, which is specified by you; and 2) conditional upon the treatment effect, the estimated effect is normally distributed. The normal prior has several advantages, including: 1) bell-shaped distribution around the prior mean, so that extreme estimates will be shrunk more towards the prior than moderate estimates; 2) the ability to specify two moments, which is often the right amount of information for a prior; and 3) simplicity. The conditional normality of the effect estimate is motivated by the central limit theorem. We use the normal distribution below to approximate the posterior: $$ \begin{align*} \Delta|\hat{\Delta} &\stackrel{}{\sim}\mathcal{N}\left(\Omega^{-1}\omega, \Omega^{-1}\right) \\\Omega &= 1/\sigma_{prior}^{2} + 1/\hat{\sigma}_{\Delta}^{2} \\\omega &= \mu_{prior}/\sigma_{prior}^{2} + \hat{\Delta}/\hat{\sigma}_{\Delta}^{2}. \end{align*} $$ This is an approximation to the posterior because $\Delta$ affects $\hat{\sigma}_{\Delta}^{2}$. We tested this approximation through extensive simulations, and found it had comparable coverage and mean squared error to a posterior distribution empirically sampled using Metropolis Hastings. We define rejection as the $100(1-\alpha)$\% confidence interval not containing zero. For our posterior approximation, this occurs if the posterior mean for $\Delta|\hat{\Delta}$ (i.e., $\Omega^{-1}\omega$) divided by its posterior standard deviation$\left(\text{i.e., }\sqrt{\Omega^{-1}}\right)$ is beyond the the appropriate critical threshold $Z^{\star}$ (e.g., $\Phi^{-1}(0.975)$ for $\alpha=0.05$). Inside of a Bayesian framework, it can help to permit the case where the prior model is misspecified. That is, the prior specified by the customer differs from the true prior that generates the treatment effect. We permit misspecification of the prior for $\Delta$, as we assume that the true data generating process (DGP) is $\Delta \stackrel{}{\sim}\mathcal{N}\left(\mu_{\star}, \sigma_{\star}^{2}\right)$, while the specified DGP has $\Delta \stackrel{}{\sim}\mathcal{N}\left(\mu_{prior}, \sigma_{prior}^{2}\right)$. We assume the prior is specified on the relative scale. In derivations below we use the marginal distribution of $\hat{\Delta}$, which we find using its moment generating function: $$ \begin{align*} E\left[\exp^{t\hat{\Delta}}\right] &= E_{\Delta}\left[E\left[\exp^{t\hat{\Delta}}|\Delta\right]\right] \\&= E_{\Delta}\left[\exp^{t\Delta + t^{2}\hat{\sigma}_{\Delta}^{2}}\right] \\&= \exp^{ t^{2}\hat{\sigma}_{\Delta}^{2}}E_{\Delta}\left[\exp^{t\Delta}\right]\ \\&= \exp^{ t^{2}\hat{\sigma}_{\Delta}^{2}}\exp^{t\mu^{\star} + t^{2}\sigma_{\star}^{2}} \\&\stackrel{}{\sim}\mathcal{N}\left(\mu^{\star}, \sigma_{\star}^{2}+\hat{\sigma}_{\Delta}^{2}\right). \end{align*} $$ For a 2-sided test the probability of rejection is $$ \begin{align*} &P\left(\left|\frac{\Omega^{-1}\omega}{\Omega^{-1/2}}\right| > Z_{1-\alpha/2}\right) \\&=P\left(\left|\left(\frac{1}{\sigma_{prior}^{2}} + \frac{1}{\hat{\sigma}_{\Delta}^{2}}\right)^{-1/2} \left(\frac{\mu_{prior}}{\sigma_{prior}^{2}} + \frac{\hat{\Delta}}{\hat{\sigma}_{\Delta}^{2}}\right)\right|> Z_{1-\alpha/2}\right) \\&= P\left(\left|\left(\frac{\mu_{prior}}{\sigma_{prior}^{2}} + \frac{\hat{\Delta}}{\hat{\sigma}_{\Delta}^{2}}\right)\right|> \left(\frac{1}{\sigma_{prior}^{2}} + \frac{1}{\hat{\sigma}_{\Delta}^{2}}\right)^{1/2} Z_{1-\alpha/2}\right) \\&=P\left(\left(\frac{\mu_{prior}}{\sigma_{prior}^{2}} + \frac{\hat{\Delta}}{\hat{\sigma}_{\Delta}^{2}}\right)> \left(\frac{1}{\sigma_{prior}^{2}} + \frac{1}{\hat{\sigma}_{\Delta}^{2}}\right)^{1/2} Z_{1-\alpha/2}\right) \\&+ P\left(\left(\frac{\mu_{prior}}{\sigma_{prior}^{2}} + \frac{\hat{\Delta}}{\hat{\sigma}_{\Delta}^{2}}\right) \hat{\sigma}_{\Delta}^{2}\left[\left(\frac{1}{\sigma_{prior}^{2}} + \frac{1}{\hat{\sigma}_{\Delta}^{2}}\right)^{1/2} Z_{1-\alpha/2} - \frac{\mu_{prior}}{\sigma_{prior}^{2}}\right]\right) \\&+ P\left(\hat{\Delta} \frac{\hat{\sigma}_{\Delta}^{2}\left[\left(\frac{1}{\sigma_{prior}^{2}} + \frac{1}{\hat{\sigma}_{\Delta}^{2}}\right)^{1/2} Z_{1-\alpha/2} - \frac{\mu_{prior}}{\sigma_{prior}^{2}}\right]-\mu_{\star}}{\sqrt{\hat{\sigma}_{\Delta}^{2}+\sigma^{2}_{\star}}}\right) \\&+ P\left(\frac{\hat{\Delta}-\mu_{\star}}{\sqrt{\hat{\sigma}_{\Delta}^{2}+\sigma_{\star}^{2}}} ![Metric Icon showing CUPED Disabled Reason](/images/statistics/power-decreasing.png) Figure 1 shows the case where the group sample sizes are 1500, the data mean is 0.1, the data variance is 0.5, the specified prior mean $\mu_{prior} = 0.1$, the specified prior variance $\sigma_{prior}^{2}$ is 0.3, and the variance of the data generating process $\sigma_{\star}^{2}$ is 0. Nominal 80% power occurs at 1.65, continues to increase in effect size until the effect size is about 3, and then begins decreasing. Power decreases in effect size because the variance in Equation 8 is quadratic in effect size, and in Equation 11 the term in front of $Z_{1-\alpha/2}$ goes to infinity as $\Delta$ gets large. The coefficient in front of $Z_{1-\alpha/2}$ in Equation 4 is 1, so frequentist power is increasing in effect size in all cases. Monotonicity does hold for Bayesian power for absolute effects, where the variance is not affected by the effect size. Because power is not monotonic in effect size, we perform a grid search across effect sizes ranging from 0 to 500%. The derivative of Equation 11 is bounded in absolute value by $2 \phi(0) 0$. Because 1) the maximum slope from the midpoint to the endpoint of the cell is no greater than $2 \phi(0)$; and 2) the maximum distance from where power is evaluated is $l/2$, the maximum power in $\left[(l-1)k, lk \right]$ is no greater than $\max\left(\pi_{k-1}, \pi_{k}\right)+\phi(0)l$. Motivated by this fact, we describe our approach in Algorithm 1. In words, Algorithm 1 evaluates power at $\Delta=\left\{0, 0.001, 0.002, ... 5\right\}$, until it finds the first element $k$ such that $\pi(k) >= \pi - \phi(0)l$. If power exceeds this threshold, then we evaluate a finer grid across the range from $[k-l, k]$, where the grid cell length is $l' = \pi - \phi(0)l$. Define $l'$ as a finer grid resolution (in production, $l' = l / 100$). Find the first element of the set $\left\{k -1, k-1+l', k-1+2l', ..., k-l', k \right\}$ such that power evaluated at that point is at least $\pi$. If no such point exists, return to the coarser grid search in Step 3. If power exceeds $\pi + \phi(0)l'$ at any point in $[0, 5]$, then Algorithm 1 is guaranteed to detect it. In practice we use $l=10^{-3}$ and $l'=10^{-5}$. --- ## Power Analysis # Power ## What is power analysis? Power analysis helps you estimate required experimental duration. **Power** is the probability of observing a statistically significant result, given your feature has some effect on your metric. ## When should I run power analysis? You should run power analysis before your experiment starts, to determine how long you should run your experiment. The longer the experiment runs, the more users are in the experiment (i.e., your sample size increases). Increasing your sample size lowers the uncertainty around your estimate, which makes it likelier you achieve a statistically significant result. ## What is a minimum detectable effect, and how do I interpret it? The relative effect of your experiment (which we often refer to as simply the effect size) is the percentage improvement in your metric caused by your feature. For example, suppose that average revenue per customer under control is \$100, while under treatment you expect that it will be \$102. This corresponds to a (\$102-\$100)/\$100 = 2% effect size. Effect size is often referred to as lift. Given the sample variance of your metric and the sample size, the minimum detectable effect (MDE) is the smallest effect size for which your power will be at least 80%. GrowthBook includes both power and MDE results to ensure that customers comfortable with either tool can use them to make decisions. The MDE can be thought of as the smallest effect that you will be able to detect most of the time in your experiment. We want to be able to detect subtle changes in metrics, so smaller MDEs are better. For example, suppose your MDE is 2%. If you feel like your feature could drive a 2% improvement, then your experiment is high-powered. If you feel like your feature will probably only drive something like .5% improvement (which can still be huge!), then you need to add users to reliably detect this effect. ## How do I run a power analysis? 1. From the GrowthBook home page, click "Experiments" on the left tab. In the top right, click "Power Calculator." 2. Select “New Calculation”. 3. On the first page you will select two different components of a power calculation: - First, the population for your power analysis. This should be the Fact Table, Segment, or Past Experiment that best represents the kind of traffic you will get in the experiment you are planning. We will use this population to estimate both your weekly traffic as well as your metric mean and variance. - Second, select your goal metrics for your experiment (maximum of 5). Only metrics that can be joined to your population of interest or that were a part of a past experiment can be selected. All non-quantile metrics are supported. 4. On the second page you specify your Expected Effect Size and can customize the traffic and metric values used in the power analysis. - For each metric, enter the "Expected Effect Size" for each metric. Effect size is the percentage improvement in your metric (i.e., the lift) you anticipate your feature will cause. Inputting your effect size can require care - see [here](#how-should-i-pick-my-effect-size). - For each metric and for the users per week, we pre-populate the data with values from your selected population, but you can also customize the, if you wish. 5. Now you have results! Please see the results interpretation [here](#how-do-i-interpret-power-analysis-results). 6. You can modify the number of variations you intend to test, as well as clicking "Edit" in the Analysis Setting box if you want to change your statistics engine or statistics settings. 7. If you want to modify your inputs, click the "Edit" button next to "New Calculation" in the top right corner. ## How do I interpret power analysis results? In this section we run through an example power analysis using our own GrowthBook app (some numbers have been modified for example purposes). For the example, we will use the Jitsu Events fact table as our population. This is basically all users that use GrowthBook regularly, so it mimics an experiment that would happen to all users as they log in to GrowthBook. We will also use the `Created a Feature` binomial metric, which is 1 for users that create a Feature in 72 hours after the are exposed and 0 otherwise. ![Select a Fact Table population and a binomial metric in the power calculator.](/images/statistics/power_1.png) When we click next, a query kicks off using the last 8 weeks of data to estimate our expected traffic to this population as well as the metric value. ![Selecting expected effect sizes and reviewing the power query results.](/images/statistics/power_2.png) As you can see, the query estimates about 2,195 users per week and a 14.41\% of those that Created a Feature in the 72 hours after first appearing in our population (this 72 hour window comes from our metric definition; these kinds of settings on your own metrics will also be applied to make the power calculation more accurate). We also selected a 20\% expected effect size. This is large, but not unattainable based on our past experiments and so it is our best guess for our expected metric improvement. We provide guidance for effect size selection [here](#how-should-i-pick-my-effect-size). For our feature we anticipate a 20% improvement in users creating a feature. We then submit our data. Now we can see the results! ![metric and users input](/images/statistics/power_3.png) At the top of the page is a box called Analysis Settings. If you want to rerun power results with number of variations greater than 2, then change the "# of Variations" and the results will automatically update. The total traffic divided by number of variations should equal the smallest sample size in the experiment you plan to run. If you want to toggle on or off "Sequential Testing", then press the "Edit" button and select the appropriate option. Enabling sequential testing reduces power. Below "Analysis Settings" is "Calculated Sample Size and Runtime", which contains the number of weeks (or equivalently the number of users) needed to run your experiment to achieve 80% power by metric. Clicking on a row in the table causes the information in the box to the right to change. We expect 80% power for our binomial metric if we run the experiment for 3 weeks. Beneath "Calculated Sample Size and Runtime" is "Power Over Time", which contains power estimates by metric. The columns in Power Over Time correspond to different weeks. For example, in the first week power is 41\%. The highlighted column in week 3 is the first week where at least 80\% power is achieved for our metric. As expected, power increases over time, as new users are added to the experiment. Beneath Power Over Time is Minimum Detectable Effect Over Time. Minimum Detectable Effect Over Time is structured the same as Power Over Time, except it contains MDEs rather than power estimates. The Week 1 MDE is 34.5\%. This means that if your true lift is 34.5\%, after 1 week you will have at least 80\% chance of observing a statistically significant result. As expected, MDEs decrease over time, as new users are added to the experiment and we have more power to detect smaller and smaller effects. If you see `N/A` in your MDE results, this means that you need to increase your number of weekly users, as the MDE calculation failed. It can be helpful to see power estimates at different effect sizes, different estimates of weekly users, etc. To modify your inputs, click the "Edit" button next to "New Calculation" in the top right corner. ## How should I pick my effect size? Selecting your effect size for power analysis requires careful thought. Your effect size is your anticipated metric lift due to your feature. Obviously you do not have complete information about the true lift, otherwise you would not be running the experiment! We advocate running power analysis for multiple effect sizes. The following questions may elicit helpful effect sizes: 1. What is your best guess for the potential improvement due to your feature? Are there similar historical experiments, or pilot studies, and if so, what were their lifts? 2. Suppose your feature is amazing - what do you think the lift would be? 3. Suppose your feature impact is smaller than you think - how small could it be? Ideally your experiment has high power (see [here](#what-is-a-high-powered-experiment)) across a range of effect sizes. ## What is a high-powered experiment? In clinical trials, the standard is 80%. This means that if you were to run your clinical trial 100 times with different patients and different randomizations each time, then you would observe statistically significant results in at least roughly 80 of those trials. When calculating MDEs, we use this default of 80%. That being said, running an experiment with less than 80% power can still help your business. The purpose of an experiment is to learn about your business, not simply to roll out features that achieve statistically significant improvement. The biggest cost to running low-powered experiments is that your results will be noisy. This usually leads to ambiguity in the rollout decision. ## How do I run Bayesian power analysis? For Bayesian power analysis, you specify the prior distribution of the treatment effect (see [here](/statistics/details#bayesian-engine)) for guidance regarding prior selection). We then estimate Bayesian power, which is the probability that the $(1 - \alpha)$ credible interval does not contain 0. If your organizational default is Bayesian, then Bayesian will be your default power analysis. You can switch to and from frequentist and Bayesian power calculations by toggling "Statistics Engine" under "Settings" on the Power Results page. Your default prior for each metric will be your organizational default. To change a prior for a metric, go to "Settings", and make sure that "Statistics Engine" is toggled to "Bayesian." Then choose "Prior Specification", and update prior means and standard deviations for your metric(s). Remember that these priors are on the relative scale, so a prior mean of 0.1 represents a 10% lift. ## FAQ Frequently asked questions: 1. How can I increase my experimental power? You can increase experimental power by increasing the number of daily users, perhaps by either expanding your population to new segments, or by including a larger percentage of user traffic in your experiment. Similarly, if you have more than 2 variations, reducing the number of variations increases power. 2. What if my experiment is low-powered? Should I still run it? The biggest cost to running a low-powered experiment is that your results will probably be noisy, and you will face ambiguity in your rollout/rollback decision. That being said, you will probably still have learnings from your experiment. 3. What does "N/A" mean for my MDE result? It means there is no solution for the MDE, given the current number of weekly users, control mean, and control variance. Increase your number of weekly users or reduce your number of treatment variations. 4. After looking at my effect estimate and its uncertainty on the GrowthBook UI, I entered them into the power calculator. While my results were statistically significant, the power calculator outputted that my power is less than 80%. Is this an error? This is not an error. Suppose your effect estimate from your experiment is 2%, and it is barely statistically significant. If you enter a 2% effect size into the power calculator (along with the sample means and standard deviations from your results), the power calculator will probably output power less than 80%. Why? Roughly speaking, the power calculator assumes you are going to run 100 experiments in the future. In some of these experiments your estimated effect size will be larger than 2%, and will probably be statistically significant. In others, the estimated effect size will be less than 2%, and may not be statistically significant. If fewer than 80 of these experiments are statistically significant, then your power estimate will be less than 80%. Similarly, if you enter the sample means and standard deviations from your results, the power calculator will probably output and MDE greater than 2%. ## GrowthBook implementation For both Bayesian and frequentist engines, we produce two key estimates: 1. Power - After running the experiment for a given number of weeks and for a hypothesized effect size, what is the probability of a statistically significant result? 2. Minimum Detectable Effect (MDE) - After running an experiment for a given number of weeks, what is the smallest effect that we can detect as statistically significant 80% of the time? Each engine arrives at these values differently. Below we describe high-level technical details of our implementation. All technical details can be found [here](/statistics/power-technical). ### Frequentist implementation Below we define frequentist power. Define: 1. the false positive rate as $\alpha$ (GrowthBook default is $\alpha=0.05$). 2. the critical values $Z_{1- \alpha / 2}= \Phi^{-1}(1-\alpha/2)$ and $Z_{1-\alpha}= \Phi^{-1}(1-\alpha)$ where $\Phi$ is the inverse cumulative distribution function of the standard normal distribution. We make the following assumptions: 1. equal sample sizes across control and treatment variations. If unequal sample sizes are used in the experiment, use the smaller of the two sample sizes. This will produce conservative power estimates. 2. equal variance across control and treatment variations; 3. observations across users are independent and identically distributed; and 4. all metrics have finite variance. 5. you are running a two-sample t-test. If in practice you use [CUPED](/statistics/cuped), your power will be higher. Use CUPED! For a 2-sided test (all GrowthBook tests are 2-sided), power is the probability of rejecting the null hypothesis of no effect, given that a nonzero effect exists. Mathematically, **frequentist power** equals $$ \begin{align} \pi = 1 - \Phi\left(Z_{1-\alpha/2}-\frac{\Delta}{\hat{\sigma}_{\Delta}}\right) + \Phi(Z_{\alpha/2} - \frac{\Delta}{\hat{\sigma}_{\Delta}}). \end{align} $$ Let $HW$ be the interval halfwidth. We can write the above equation as $$ \begin{align} \pi = 1 - \Phi\left( \frac{HW-\Delta}{\hat{\sigma}_{\Delta}} \right) + \Phi\left( \frac{-HW-\Delta}{\hat{\sigma}_{\Delta}} \right) \end{align} $$ The MDE is the smallest effect size for which nominal power (i.e., 80%) is achieved. In the 2-sided case there is no closed form solution. The **frequentist MDE** is the solution for $\Delta$ when solving for $\pi$ in the equation below: $$\pi = 1 - \Phi\left(Z_{1-\alpha/2}-\frac{\Delta}{\hat{\sigma}_{\Delta}}\right).$$ Inverting this equation requires some care, as the uncertainty estimate $\hat{\sigma}_{\Delta}$ depends upon $\Delta$. Details can be found [here](/statistics/power-technical). To estimate power under [sequential testing](/statistics/sequential), we adjust the variance term $\hat{\sigma}_{\Delta}^{2}$ to account for sequential testing, and then input this adjusted variance into our power formula. We assume that you look at the data only once, so our power estimate below is a lower bound for the actual power under sequential testing. Otherwise we would have to make assumptions about the temporal correlation of the data generating process. ### Bayesian implementation For Bayesian power analysis, we let users specify the prior distribution of the treatment effect (see [here](/statistics/details#bayesian-engine) for guidance regarding prior selection). We then estimate Bayesian power, which is the probability that the $(1 - \alpha)$ credible interval does not contain 0. At GrowthBook, **Bayesian power** is $$ \begin{align} \begin{split} \pi &= 1-\Phi\left( \frac{ \hat{\sigma}_{\Delta}^{2}\left[\left(\frac{1}{\sigma_{prior}^{2}} + \frac{1}{\hat{\sigma}_{\Delta}^{2}}\right)^{1/2} Z_{1-\alpha/2} - \frac{\mu_{prior}}{\sigma_{prior}^{2}}\right]-\Delta} {\sqrt{\hat{\sigma}_{\Delta}^{2}}} \right) \\&+\Phi\left( \frac{ -\hat{\sigma}_{\Delta}^{2}\left[\left(\frac{1}{\sigma_{prior}^{2}} + \frac{1}{\hat{\sigma}_{\Delta}^{2}}\right)^{1/2} Z_{1-\alpha/2} - \frac{\mu_{prior}}{\sigma_{prior}^{2}}\right]-\Delta} {\sqrt{\hat{\sigma}_{\Delta}^{2}}} \right). \end{split} \end{align} $$ Constructing the MDE is less straightforward, as MDEs are not well defined in Bayesian literature. We provide MDEs in Bayesian power analysis for customers that are used to conceptualizing MDEs and want to be able to leverage prior information in their analysis. We define the MDE as the minimum value of $\Delta$ such that at least $\pi$ power is achieved. The **Bayesian MDE** is the solution for $\Delta$ when solving for $\pi$ in Equation 1. To find $\Delta$, we use a grid search. --- ## Quantile Testing import CommercialFeature from '@site/src/components/CommercialFeature'; # Quantile Testing :::note Quantile Testing is incompatible with Mixpanel or MySQL integrations. ::: ## What is a quantile test? Quantile tests, also known as percentile tests, compare quantiles across variations. In contrast, standard GrowthBook A/B tests compare means across variations. For example, suppose that treatment increases user spend. Suppose that 90% of users in control spend at most \$50 per visit, and 90% of users in treatment spend at most \$55 per visit. Then the quantile treatment effect at the 90th percentile (i.e., P90) is \$55 - \$50 = \$5. Quantiles are commonly used in many applications where very large or small values are of interest (webpage latency, birth weight, blood pressure). ## When should I run quantile tests? Below are two scenarios where you should run quantile tests. Scenario 1: your website has low latency for most users, but for 1% of users it takes a long time for the page to load. You have a potential solution that targets improvements for this small fraction, and run an A/B test to confirm. Mean differences across variations may be noisy and provide uncertain conclusions, as your solution does not improve latency for most users. Running a quantile test at the 99th percentile can be more informative, as it helps detect if users that would have experienced the largest latency had their latency reduced. Scenario 2: you have a new ad campaign designed to increase customer spend. You run an A/B test, and while the lift is positive, it is not statistically significant. Quantile tests can help you deep dive which subpopulations were positively affected by treatment. For example, you may see no improvement at P50, but moderate improvement at P99. This would indicate that the new campaign did not affect most users, but had a strong affect on your top spending users. In summary, quantile tests can complement mean tests. ## How do I interpret quantile test results? Suppose you are running an A/B test designed to reduce website latency. Your effect estimate and 95% confidence interval (in milliseconds) for latency at P99 is -7 ms (-9 ms, -5 ms). How do you interpret this? Consider one universe where **all** customers received control (not just the customers assigned to control). In this universe, website latency for 99% of customers is no more than 145 ms. That is, P99 for control is 145 ms. Consider another universe where **all** customers are assigned to treatment. In the treatment universe, website latency for 99% of customers is no more than 139 ms. So the true effect at P99 is 139 ms - 145 ms = -6 ms. A quantile test tries to estimate this difference. You interpret the interval above as “There is a 95% chance that the difference in P99 latencies across the groups is in (-9ms, -5ms)”. ## Should I aggregate by experiment user before taking quantile? Below we describe how quantile testing differs in event- vs user-level analyses. Suppose that your new feature is designed to lower webpage request times. If you want to reduce the largest request times across all web sessions, then use quantile testing for event-level data. This can help you learn if your new feature reduced the 99th percentile of request times. If you want to reduce total request times for your most frequent customers, then use quantile testing for user-level data. Here GrowthBook sums the total request times for all events within a customer, then compares percentiles of these sums across variations. Sometimes it can be hard to choose whether or not to aggregate. Another consideration is how the metric is typically conceptualized. Aggregation is usually correct if your metric is often conceptualized at the customer level (e.g., customer spend during a week). Aggregation is usually inappropriate if your metric is often conceptualized at the event level (e.g., spend per order). To illustrate the mathematics behind the two approaches, consider an experiment where we collected the following data for all users in a variation. | user_id | value | | ------- | ----- | | 123 | 0 | | 123 | 0 | | 456 | 2 | | 456 | 3 | | 789 | 99 | Taking the median (P50) under each approach gives different results: | Quantile Type | P50 | Formula | | ------------- | --- | ------------------------------------------ | | Event | 2 | `APPROX_PERCENTILE([0, 0, 2, 3, 99], 0.5)` | | Per-User | 5 | `APPROX_PERCENTILE([0, 5, 99], 0.5)` | Notice that for the Per-User quantile, we sum values at the user level first (user 123: 0, user 456: 5, user 789: 99) before passing them to the percentile function. `NULL` values are always ignored. ## How do I run a quantile test in GrowthBook? Running a quantile test is just as easy as running a mean test. 1. Navigate to your [Fact Table](/app/metrics) and select "Add Metric". 2. Select “Quantile” from “Type of Metric”. 3. Toggle “Aggregate by Experiment User before taking quantile” if you want to compare quantiles across variations at the user granularity, after summing row values at the user level. The default is at the event granularity. 4. Pick your quantile level from the defaults (p50, p90, p95, p99) or use a custom value. Guidance describing the range of available values is in our [FAQ](#faq) at the bottom of this page. 5. Decide whether you want zeros to be included in the analysis (see [FAQ](#faq) at the bottom of this page). 6. Select your metric window as you would for a mean test. 7. Submit! ![User interface for quantile metrics](/images/statistics/quantile.png) ## GrowthBook implementation Technical details Here we describe technical details of our implementation. GrowthBook implements the approach first introduced in [Deng, Knoblich and Yu (2018)](https://alexdeng.github.io/public/files/kdd2018-dm.pdf). This clever approach has two key advantages. First, it constructs valid confidence intervals for quantiles that uses only sample quantiles, rather than all of the data. This permits estimation using only a single pass through the data. Second, it provides quantile inference for clustered data. This is helpful when randomization occurs at the user level, but our metrics are measured at the session level (described [here](#should-i-aggregate-by-experiment-user-before-taking-quantile)). Our implementation is based upon Algorithm 1 of [Yao, Li and Lu (2024)](https://arxiv.org/pdf/2401.14549.pdf). Define $\nu \in (0, 1)$ as the quantile level of interest. Define $\alpha \in (0,1)$ as the false positive rate, and let $Z_{1-\alpha/2}$ be its associated critical value. Without loss of generality we focus on the control variation. Let $n$ be the control sample size. Define $Y_{ij}$ as the webpage latency for the $j^{\text{th}}$ session for the $i^{\text{th}}$ user (i.e., cluster) in control, $j=1,2,…, N_{i}$, $i=1,2,…,K$. Define the observed control outcomes as $\left\{Y_{1}, Y_{2}, ..., Y_{n}\right\}$, where $n=\sum_{i=1}^{K}N_{i}$. Define the ordered (from smallest to largest) control outcomes as $\left\{Y_{(1)}, Y_{(2)}, ..., Y_{(n)}\right\}$. 1. Compute L, U = $n\left(\nu \pm Z_{1-\alpha/2}\sqrt{\nu(1-\nu)/n} \right)$. 2. Fetch $Y_{n\nu}, Y_{L}, Y_{U}$. 3. Compute $I_{ij} = 1\left\{Y_{ij}\leq Y_{n\nu}\right\}$. Define $\bar{I} = n^{-1}\sum_{i=1}^{K}\sum_{j=1}^{N_{i}}I_{ij}$. 4. Compute $\sigma_{I, \text{iid}}^{2} = \nu(1-\nu)/n$, an estimate of the variance of $\bar{I}$ assuming independent and identically distributed (iid) errors. 5. Define $\sigma_{I, c}^{2} = \text{Var}(\bar{I})$ using the variance of ratios of means described below. 6. Compute $\sigma_{iid}^{2} = \left(\frac{Y_{U}-Y_{L}}{2Z_{1-\alpha/2}}\right)^{2}$ . 7. The cluster-adjusted variance is $\sigma_{\text{iid}}^{2} \left(\sigma_{I,\text{iid}}^{2}/\sigma_{I,c}^{2}\right)$. The term in parentheses adjusts the variance for clustering. If there is no clustering (i.e., inference is at the user level), then use $\sigma_{iid}^{2}$. To further speed this algorithm, instead of finding the exact $(L, U)$, which requires pre-computing $n$ inside of SQL, we instead construct a sequence of logarithmically increasing sample sizes $N^{\star}=\left\{n_{1}, n_{2}, ... n_{M}\right\}$ and their associated intervals$\left\{(L_{1}, U_{1}), (L_{2}, U_{2}), ..., (L_{M}, U_{M}) \right\}$. We then output the quantiles associated with each interval, as well as $n$. We then find $n^{\star}$, defined as the largest element of $N^{\star}$ such that $n^{\star}\leq n$. We then adjust the variance outputted by the algorithm above by a factor of $n^{\star}/n$. Currently we use 20 different values of $N^{\star}$, where the $k^{\text{th}}$ value is equal to $100 * 2 ^{k-1}$. In this paragraph we describe the variance of a ratio of means, as described in step 5. above. Additionally, this is the standard formula used to calculate variance of a mean for a variation in GrowthBook t-tests. Here we define the user outcome in terms of random variable $X_{i}$, as the formula below can be used for any outcome (e.g., $Y_{ij}$, $I_{ij}$, etc.). For the $i^{\text{th}}$ user define the sum of outcomes $S_{i} = \sum_{j=1}^{N_{i}}Y_{ij}$. Then the mean outcomes across users is $\bar{X}=\frac{\sum_{i=1}^{K}S_{i}}{\sum_{i=1}^{K}N_{i}}$. Define the mean sum of latencies across users as $\bar{S} = K^{-1}\sum_{i=1}^{K}S_{i}$. Define the mean sum of sessions across users as $\bar{N} = K^{-1}\sum_{i=1}^{K}N_{i}$. A formula for the variance of $\bar{X}$ is $\text{Var}\left(\bar{X}\right) = \frac{1}{K\bar{N}^{2}}\left[\text{Var}(S)-2\frac{\bar{S}}{\bar{N}}\text{Cov}(S,N)+\frac{\bar{S}^2}{\bar{N}^2}\text{Var}(N)\right]$. This is similar to the delta method approximation of the variance we use for ratio metrics and for relative effects. Let $\hat{\mu}_{C,n\nu}$ be the $\nu^{\text{th}}$ sample quantile for control, and let its associated variance be $\hat\sigma^2_{C,n\nu}$. Analogously define $\hat{\mu}_{T,n\nu}$ and $\hat\sigma^2_{C,n\nu}$ for treatment. These quantities are plugged into our lift estimators as described in the [Statistical Details](/statistics/details) page. The result from step 7 is our estimate of the variance and the sample quantile is directly computed in the SQL query. ## FAQ Frequently asked questions: 1. Can I pick any quantile level (e.g., P99.999)? No - the maximum range available is [0.001, 0.999], and that is only for experiments with large sample sizes (i.e., n > 3838). This is because inference can become unreliable for extreme quantile levels and small sample sizes. In general, if you want to compare quantiles at some value $p \in (0, 1)$, and you want a 95% confidence interval, your sample size $n$ must be bigger than $4p/(1-p)$. Similarly, if you want extreme and small quantiles, you need $n \geq 4(1-p)/p$. For $p=0.99$ and $p=0.01$, this corresponds to $n \geq 380$. 2. Should I include zeros in my quantile test? This depends upon the population that you care about, and what you want to learn. If zero is a common value for your metric, then P90 including zeros can be much less than P90 without zeros. If your metric is typically conceptualized and reported with zeros included, then it will probably makes sense to include zeros in your quantile test metric. If you are using quantile tests to deep dive mean test results, then use the same configuration for both tests. 3. Can I get quantile test results inside of a Bayesian framework? Yes - GrowthBook puts a prior on the quantile treatment effect, and combines this prior with the effect estimate to obtain a posterior distribution for the quantile treatment effect. So “Chance to Win” and other helpful Bayesian concepts are available. 4. How does quantile inference connect to mean inference? If you average the quantiles of a distribution, you get the distributional mean. That is, the average of $\left\{P1, P2, ..., P98, P99\right\}$ equals the mean of the distribution. Similarly, the average of the treatment effects at P1, P2, etc. equals the mean treatment effect. So quantile inference can be viewed as a decomposition of mean inference. 5. How should I conduct quantile inference in the presence of percentile capping? We have disabled percentile capping for quantile testing. For example, if you picked P99 for your quantile level, then your results could be biased, as capping at P98 ignores all information beyond the 98th percentile. Percentile capping at P98 does not affect estimates at any quantile level below P98 (e.g., P50, P90), so percentile capping will either do nothing or potentially bias quantile test results. 6. How does Quantile Testing intersect with [CUPED](/statistics/cuped), [Multiple Testing Corrections](/statistics/multiple-corrections), and [Sequential Testing](/statistics/sequential)? Currently CUPED and Sequential Testing are not implemented for Quantile Testing. Multiple Testing Corrections is implemented for Quantile Testing. 7. What is cluster adjustment? Data are clustered when randomization happens at a coarser granularity than the metrics of interest. For example, suppose we are trying to reduce webpage latency. We randomize customers (perhaps due to engineering constraints or testing purposes). A customer may have multiple sessions, i.e., a session is clustered within customer. Latencies for two sessions from the same customer are likelier to be more similar than latencies for two sessions from different customers. Cluster adjustment ensures that we do not overstate the amount of information in the experiment, i.e., uncertainty estimates are valid. 8. Are quantile estimates available via [incremental refresh](/app/data-pipeline#incremental-refresh-recommended-in-beta)? Yes - event-level quantile metrics are supported for BigQuery, which uses KLL sketches to store approximate event-level data for each experiment unit. This provides approximate inference, as small effects may be undetectable. --- ## Sequential Testing import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; # Sequential Testing :::info Sequential Testing is only implemented for the Frequentist statistics engine. ::: ## Why use sequential testing? Sequential testing allows you to look at your experiment results as many times as you like while still keeping the number of false positives below the expected rate. In other words, it is the frequentist solution to the peeking problem in AB testing. Sequential testing not only reduces the risk of false positives in online experimentation; it can also increase the velocity of experimentation. Although sequential testing produces wider confidence intervals than fixed-sample testing, traditional frequentist inference requires an experimenter to wait until a pre-determined sample size is collected. With sequential testing, decisions can be made as soon as significance is reached, without fear of inflating the false positive rate. ### The peeking problem What is peeking? First, some background on frequentist testing. When running a frequentist analysis, the experimenter sets a confidence level, often written as $\alpha$. Many people, and GrowthBook, default to using $\alpha = 5\%$ (sometimes it is written as $\alpha = 0.05$). These all mean the same thing: for a correctly specified frequentist analysis, you will only reject the null hypothesis 5% of the time when the null hypothesis is true. Usually in online experimentation, our null hypothesis is that metric averages in two variations are the same. In other words, $\alpha$ controls the _False Positive Rate_. However, experimenters often violate a fundamental assumption underpinning frequentist analysis: that one must wait until for some pre-specified time (or sample size) before looking at and acting upon experiment results. If you violate this assumption, and _peek_ at your results, you will end up with an inflated _False Positive Rate_, far above your nominal 5% level. In other words, if we check an experiment as it runs, we are essentially increasing the number of times we can get a positive, even if there is no experiment effect, just through random noise. We have two options: 1. Stop checking early. This is possible, but in a lot of cases it can make decision making worse! It is powerful to be able to see bad results and shut a feature down early; or conversely to see a great feature do well right away, end the experiment, and ship to everyone. 2. Use an estimator that returns our control over false positive rates. Sequential testing provides estimators for the second option; it allows peeking at your results without fear of inflating the false positive rate beyond your specified $\alpha$. ## Configuring sequential testing You can enable or disable sequential testing, as well as select your tuning parameter, both in your organization settings (as a default for all future experiments) or on an individual experiment's page. You can also play with the settings in a custom report. ### Setting the tuning parameter The tuning parameter (written as $N^*$ in the [implementation notes below](#growthbooks-implementation)), should be set to the number of observations at which you tend to make decisions. It can be thought of as an "estimated sample size" for the two arms you are comparing. You want to set $N^*$ to be the number of observations at which you tend to make decisions, because it is at that sample size that the loss in experiment power from using sequential testing will be smallest relative to the traditional confidence intervals. The following figure shows how the increased CI width in sequential analysis is minimized when the sample size is approximately $N^*$. The y-axis is the ratio of the sequential CIs to the regular CIs; all lines are well above 1.0, showing that sequential analysis results in uniformly wider confidence intervals. On the x-axis is the sample size at which the analysis was executed. As you can see, the ratio is lowest when $N^*$ is as close to sample size used in the analysis. ![Effect of Tuning Sequential Statistics on CI Width](/images/statistics/sequential-tuning.png) We default to using 5,000, but the general advice is to set this parameter to the sample size you expect to get when you are most likely to make a decision on this experiment. Note, this number reflects the total sample size across the two variations you are comparing. If you have an experiment with 3 variations, and you want the smallest confidence intervals after 3,000 users in each variation, you should set the tuning parameter to 6,000 following the above logic. This is because while the total number of users will be 9,000, there will be 6,000 total in each comparison of arms (e.g. A vs B, B vs C, and A vs C). Note that the tuning parameter should remain fixed for an experiment. ### Organizational defaults To set the default usage of sequential testing and the tuning parameter for new experiments, navigate to the Organization Settings page and select your preferred defaults. ![Sequential Statistics Organization Settings](/images/statistics/sequential-settings.png) ### Experiment settings You can also turn sequential testing on and off for an individual experiment, as well as set a specific tuning parameter for that experiment. Navigate to the Experiment page, click Edit Experiment Settings, and choose your preferred settings. ![Sequential Statistics Experiment Settings](/images/statistics/sequential-exp-settings.png) ## GrowthBook's implementation There are many approaches to sequential testing, several of which are well explained and compared in [this Spotify blogpost](https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions/). For GrowthBook, we selected a method that would work for the wide variety of experimenters that we serve, while also providing experimenters with a way to tune the approach for their setting. To that end, we implement Asymptotic Confidence Sequences introduced by [Waudby-Smith et al. (2023)](https://arxiv.org/pdf/2103.06476v7.pdf); these are very similar to the Generalized Anytime Valid Inference confidence sequences described by Spotify in the above post and introduced by [Howard et al. (2022)](https://arxiv.org/pdf/1810.08240.pdf), although the Waudby-Smith et al. approach more transparently applies to our setting. Specifically, GrowthBook's confidence sequences, which take the place of confidence intervals, become: $$\left(\hat{\mu} \pm \hat{\sigma}*\sqrt{N}*\sqrt{\frac{2(N\rho^2 + 1)}{N^{2}\rho^2}\log\left(\frac{\sqrt{N\rho^2 + 1}}{\alpha}\right)}\right)$$ $$\rho = \sqrt{\frac{-2\text{log}(\alpha) + \text{log}(-2 \text{log}(\alpha) + 1)}{N^*}}$$ In the above, $\hat{\mu}$ is our estimated uplift, $\hat{\sigma}$ is the estimated standard error of the uplift, $\alpha$ is our significance level (defaults to 0.05), and $N$ is the sum of the two variation's sample sizes. In the above $N^*$ is a tuning parameter that lets us control how tight these confidence sequences are relative to the standard, fixed-time confidence intervals. You can read more about how to set it in the [Configuring Sequential Testing](#setting-the-tuning-parameter) section. --- ## Sticky Bucketing import CommercialFeature from '@site/src/components/CommercialFeature'; This article serves two purposes: - High level overview of GrowthBook's Sticky Bucketing feature - Technical details on how to implement Sticky Bucketing in your codebase Sticky bucketing ensures users continue to see the same variation when you make changes to a running experiment. GrowthBook's flavor of sticky bucketing has a few additional features:(1) Bucketing based on either a primary hash attribute (i.e. user id) or a secondary attribute (i.e. anonymous id) and (2) the ability to version-control and purge your users' assigned buckets. ## Motivation So why would you want to use Sticky Bucketing? Let's look at a few examples: 1. You are managing an experiment rollout and need to slow down enrollment. You decrease the percentage of traffic exposed to the experiment from 50% to 10% but you do not want to alter the experience for users who were already exposed to the experiment. Sticky bucketing allows you to apply the rollout percentage to new users while keeping the old users in their original buckets. 2. You discovered a bug in your experiment the day after launching. You fix the bug and want to re-start the experiment, but don't want to include any "tainted" users who saw the buggy version since their negative experience might impact the results. 3. Your have a cross-platform app and want to ensure a consistent experience as users log in and move between devices. ## Setting up Sticky Bucketing with the Script Tag SDK ### 1. Configure the script tag Using Sticky Bucketing with our [HTML Script Tag](/lib/script-tag) is simple. Add either of these properties to your script tag: - `data-use-sticky-bucket-service="cookie"` or - `data-use-sticky-bucket-service="localStorage"` The sticky bucket assignments will be stored in either cookies or browser LocalStorage, respectively. Both of these implementations are session-based, but cookies are transportable and are generally more commonly used. For example: ```html ``` Further details are available [here](/lib/script-tag#sticky-bucketing). ### 2. Enable Sticky Bucketing for your organization In the GrowthBook app, go to **Settings** → **General** → **Experiment Settings** and enable the Sticky Bucketing toggle. This will add new options specific to Sticky Bucketing whenever you make changes to a running experiment. To read more about these options, see [Experiments (setup)](/app/experiment-configuration). Once Sticky Bucketing is enabled, there is an additional toggle for enabling a **Fallback Attribute**. See below for more information on this feature. ## Setting up Sticky Bucketing with other SDKs To use Sticky Bucketing for your experiments, there are a few steps that you need to complete. ### 1. Ensure you are using a compatible SDK version Update your codebase to use a compatible SDK. Sticky Bucketing is currently supported most SDK languages. See "stickyBucketing" availability in the [list of SDKs and Capabilities](https://github.com/growthbook/growthbook/blob/main/packages/shared/src/sdk-versioning/CAPABILITIES.md). ### 2. Pass a Sticky Bucketing Service into your SDK Implementation You may use one of our built-in Sticky Bucketing Services or implement your own. We provide common drivers for browser-generated cookies, backend-generated cookies, browser LocalStorage, and Redis stores. For more information about setting up Sticky Bucketing at the SDK level, see the appropriate SDK documentation. For instance, see the [Javascript SDK - Sticky Bucketing documentation](/lib/js#sticky-bucketing) for more information about setting up Sticky Bucketing in the Javascript SDK. ### 3. Update your SDK Connections in the GrowthBook app Within the GrowthBook app, ensure that your SDK Connections are configured correctly. You can do this by going to the **SDK Connections** page, clicking into a connection, and clicking the **Edit** button in the top right. Make sure the connection (1) only has a single language selected and (2) has the correct SDK version specified. ![Setting the SDK Connection Version](/images/sdk-connection-version-sb.png) In the above example, React is selected and the version is set to the latest `0.22.0`, which supports sticky bucketing. If you are using GrowthBook with multiple languages, create a separate SDK Connection for each language. ### 4. Enable Sticky Bucketing for your organization In the GrowthBook app, go to **Settings** → **General** → **Experiment Settings** and enable the Sticky Bucketing toggle. This will add new options specific to Sticky Bucketing whenever you make changes to a running experiment. To read more about these options, see [Experiments (setup)](/app/experiment-configuration). Once Sticky Bucketing is enabled, there is an additional toggle for enabling a **Fallback Attribute**. See below for more information on this feature. ## Fallback Attribute Users are assigned an experiment variation based on a **Hash Attribute**, for example a logged-in `userId`. With Sticky Bucketing, you also have the option of specifying a **Fallback Attribute** for an experiment, for example an anonymous `cookieId`. This fallback will be used if the primary hash attribute is missing or empty. ### Fallback Attribute Example Imagine your users tend to sign in on multiple devices. Let's say you want to test changes to the main navigation header of your app, something that is visible to both logged-in and anonymous visitors. If you were to only use `userId` to assign variations, signing in could become a jarring experience - users might flip from seeing the control (since their user id is empty) to seeing the variation (once they log in). On the plus side, if users open your app on multiple devices (when logged in), they will always see a consistent experience. If instead, you were to only use the anonymous `cookieId` to assign variations, it solves the issue where the UI flips during sign in (since the anonymous id stays the same before and after), but now switching devices could become a jarring experience - the same user might get assigned different variations on different devices, since each device would have its own separate anonymous id. Fallback attributes, when implemented properly with sticky bucketing, lets you have the best of both worlds (with some caveats). Your primary hashing attribute would be the logged-in `userId` and your fallback attribute would be the anonymous `cookieId`. The very first variation a user is assigned to will "stick" to them and follow them across devices. So if a visitor lands on your website, gets assigned variation B (from their fallback attribute), and then logs in, they will continue seeing variation B, even though they now have a `userId` attribute. If that same user then logs into your app on a new device, they again will continue seeing variation B. There are 2 caveats with fallback attributes: 1. There are still some scenarios where users will get inconsistent experiences. For example, if they are logged out on two devices, there's no way for us to know they are the same person. 2. It opens you up to potential bias in your experiment results (see more below). ### Bias Risk To understand the risk of bias, lets focus on a user switching devices. During analysis, we will have to use the anonymous `cookieId` as the experimental unit to make sure we capture everyone in the experiment, even those who never logged in. When a user logs in on two devices, they will be seen as two separate "users" in the analysis since each device has its own cookie id. Because of the fallback attribute and sticky bucketing, however, they will both get assigned the same variation. This breaks one of the statistical assumptions of A/B testing - that each user is randomly assigned a variation. Let's see how this might play out to cause bias in your results. Imagine your variation causes people to use multiple devices more often than your control does. 200 people land on your website and get split into control and variation - 100 in each. In the control, 20 of them also log in on their phone, but in the variation 60 of them log in on their phone. In your analysis, you have 280 total anonymous ids and you expect them to be split evenly - 140 each. In reality, the control would have 120 ids while the variation has 160. A difference this extreme is easy to spot in the results (GrowthBook runs Sample Ratio Mismatch tests automatically to catch exactly this type of bug), however there are many similar, but more subtle, issues that may fly under the radar. Bottom line: with Fallback Attributes, you can get a more consistent within-session and cross-device user experience at the expense of statistical rigor. With GrowthBook, we let you decide this trade-off for yourself on a per-experiment basis. ## Example Sticky Bucket Implementations ### Front-end only Suppose your website integrates GrowthBook on the front end only. You would like to implement Sticky Bucketing to protect against variation hopping should targeting or rollout rules change in the future. In our JavaScript and React SDKs, we provide 2 different Sticky Bucket Services that make sense in this scenario: `LocalStorageStickyBucketService` and `BrowserCookieStickyBucketService`. You can instantiate either of these services and plug them into the GrowthBook SDK. ### Front-end and Back-end (Node.js) Let's expand the "front-end only" example above so that our back-end controllers also integrate with GrowthBook and can reference the same experiments. In this scenario, we would like both the front-end and back-end to perform bucketing and persist a sticky bucket that reliably crosses the front-end / back-end divide. On the front end, you will want to use the `BrowserCookieStickyBucketService` because cookies are easily transportable to and from the back end. Then, assuming we are using an Express (NodeJS) server, we would use the `ExpressCookieStickyBucketService` on the back end. Importantly, if customizing the cookie name, you must ensure that the same name prefix is chosen for both the front-end and back-end cookies. ### Back-end only Suppose that in a server-side context we are interested in persisting a user's bucket both across multiple requests and across other back-end (micro)services that may not have direct access to the incoming user request nor their cookies. We could use a Redis instance inside our network and read/write to that for sticky bucket storage. In a NodeJS context, we could use the `RedisStickyBucketService` and pass in an `ioredis` client. ### Hybrid and custom implementations You may wish to employ multiple strategies at once (front-end, back-end, Redis) or write your own sticky bucket connector for a SQL server or DynamoDB cluster. You could write your own custom sticky bucket connector by implementing the `StickyBucketService` interface. Within your connector, you could do things like: - Connect to SQL server for sticky bucket reads/writes - GET/POST/RPC to a custom bucketing microservice - Wrap both the `ExpressCookieStickyBucketService` and `RedisStickyBucketService` within your custom service's getter and setter methods - Trigger side effects on bucket reads/writes --- ## GrowthBook DevTools - Browser Extension for Feature Flags & A/B Testing import MaxWidthImage from "@site/src/components/MaxWidthImage"; GrowthBook DevTools is a browser extension for [Chrome](https://chromewebstore.google.com/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/) that helps you inspect, debug, and modify feature flags and experiments in real time. Whether you're troubleshooting an experiment or tweaking a feature flag, DevTools gives you full visibility and control—right in your browser. ## Requirements GrowthBook DevTools is compatible with Chrome (including Edge, Arc, and most other browsers that enable Chrome extensions) and Firefox. It works with most GrowthBook client SDKs, including the HTML Script Tag, JavaScript SDK, and React SDK (client-rendered). Before using DevTools, ensure your SDK is configured with: ```bash enableDevMode: true ``` This allows DevTools to connect and display feature flags, experiments, and attributes. ## Installation The GrowthBook DevTools browser extension is available for both Chrome and Firefox. GrowthBook DevTools is available for Chrome on the [Chrome Web Store](https://chromewebstore.google.com/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) and for Firefox on [addons.mozilla.org](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/). To install: - Click **Add to Chrome** / **Add to Firefox** - Confirm the installation when prompted Once installed, GrowthBook DevTools automatically detects and connects to the GrowthBook SDK whenever it’s active on a website ✨ ## Accessing GrowthBook DevTools GrowthBook DevTools is accessible in two ways. Use whichever best fits your workflow: ### 1. Quick Access A lightweight view for quick flag and experiment testing. - Pin the extension to your toolbar (click the puzzle piece 🧩, then pin GrowthBook). - Click the GrowthBook icon to open the popup. :::tip A green dot on the GrowthBook icon means DevTools are successfully connected. ::: ![Quick access devtools](/images/devtools-quickaccess.webp) ### 2. Developer Tools Panel Ideal for when you need more space or want to inspect feature flags alongside other browser developer tools. - Open the Developer Tools with `Cmd` + `Option` + `I` (Mac) or `Ctrl` + `Shift` + `I` (Windows/Linux). - Click the **GrowthBook** tab in the Developer Tools panel. (If it’s hidden, check the overflow menu ».) ![Chrome devtools panel](/images/devtools-panel.webp) ## Essential Features GrowthBook DevTools provides powerful debugging and testing tools, including: - A real-time view of active **feature flags** and **experiments** - The ability to **override flags and experiment variations** - A detailed **debug log** explaining how and why flags were evaluated - Insights into **attributes** affecting feature rollouts - Syncing **attributes and Archetypes** directly from GrowthBook ### Feature Flags & Overrides The **Features** tab provides an overview of your feature flags, including their current values and associated experiments (indicated by the flask icon). Click on a feature to open its detail view, where you can: - Override the current value (flip a boolean, change a string, etc.). - See why the flag was evaluated the way it was under **Rules and Values**. :::tip An orange dot means an override is active. ::: ![Feature flags and overrides](/images/devtools-features.png) :::note Hiding inactive feature flags All feature flags are shown, even if they weren’t evaluated on the page. Use the **Hide inactive** toggle to hide these flags. ::: **Feature flag data table** | Field | Description | | ----------------------- | ---------------------------------------------------------------------------------------- | | Current value | The feature flag’s current value. Click or type to override. | | Rules and Values | All the rules for the flag. Defaults to showing the rule that defines the current value. | | Debug log | The reason why the flag was evaluated as such. | | Full feature definition | The complete JSON definition of the feature flag. | ### Experiments The **Experiments** tab gives a complete overview of your A/B tests, including how they’re implemented (feature flag, URL redirect, visual editor) and their current values. Click on an experiment to open its detail view, where you can: - See additional experiment data. - Override the assigned variation to test different experiences. ![Experiments overview](/images/devtools-experiments-overview.png) **Experiment data table** | Field | Description | | -------------------------- | --------------------------------------------------------------------------------- | | Enrollment Status | Whether the user is included in the experiment (irrespective of control/variant). | | Current variation | Which the user is assigned to. Click another variation to override it. | | Results log | The experiment result object. | | Current value | The value associated with the current variation. | | Implementation | How the experiment is implemented: feature flag, visual editor, URL redirect | | Targeting and Traffic | Targeting conditions, traffic/enrollment settings, and URL rules if applicable. | | Debug log | The reason why the experiment was evaluated as such. | | Full experiment definition | The complete JSON definition of the experiment. | ## Attributes Experiments and feature flags often depend on **user attributes** (e.g., location, subscription tier). The **Attributes** tab lets you: - View and edit the attributes the GrowthBook SDK is using. - Override attributes to test different scenarios (e.g., simulate a Pro user vs. Free user). - Use **JSON input** for more complex setups. ![Attributes overview](/images/devtools-attr.webp) ## Event Logs When configured in the SDK, Event logs show 3 types of data: - Feature result logs via [`onFeatureUsage`](https://docs.growthbook.io/lib/js#feature-usage-callback). - Experiment tracking callbacks via [`trackingCallback`](https://docs.growthbook.io/lib/js#feature-flag-experiments). - Metric events sent via `eventLogger`. ## SDK The SDK tab provides helpful data about your SDK, including its connection status, along with debug info and resource links if you encounter an issue. Click on each property for additional details and a short description of its function. Two particularly useful fields for debugging: 1. **Sticky Bucketing** detail shows all sticky buckets for the current user. 2. **SDK Payload** shows the actual payload the SDK is using. ![SDK overview](/images/devtools-sdk-status.webp) ## Advanced Features GrowthBook DevTools works great out of the box, but if you want even more power, enable advanced features with an Access Token: - Synced attributes: Brings in all possible attributes and their types from GrowthBook. - [Archetypes](https://docs.growthbook.io/features/rules#archetype): Toggle between preset user attributes defined in GrowthBook. - Feature revisions and drafts: Preview how feature drafts or previous revisions would behave on your site. To enable these features: - Click on the cog icon ⚙️ - Add a Personal Access Token. (Use the **Access Token** link to create a token in GrowthBook.) - If self-hosting, uncheck **GrowthBook Cloud** and enter your API details. - Click **Save**. ![DevTools Access Token](/images/devtools-pat.webp) Once saved, open the **Current User** dropdown to apply archetypes to feature flags and experiments. For example, use a **Pro user** archetype to test how your site behaves with premium features unlocked, or a **New user** archetype to see the experience for first-time visitors. ## Back End Debugging DevTools supports back end debugging. This includes: 1. Applying feature, experiment, and attribute overrides to a back-end SDK, and 2. Displaying back-end evaluation results within DevTools ![Back-end evaluation logging with DevTools](/images/devtools-back-end-evaluation.png) While much of the communication is automatically handled (mediated by cookies and datalayer-like events), some manual configuration is required to enable back end debugging. In the JavaScript SDK, we provide several environment-specific SDK plugins and helper functions to make back-end debugging simpler. See the Node.js [DevTools for back-end](/lib/node#devtools-for-back-end) guide for integration instructions. Similar approaches will work for non-JS SDKs, but they must be manually implemented. ## Visual Editor The [GrowthBook Visual Editor](https://chromewebstore.google.com/detail/growthbook-visual-editor/nbomejknbpkcpjdagefhichaajpoempk) is a separate Chrome extension for designing A/B tests without writing code. See the [Visual Editor docs](/app/visual) to install, connect, and get started. --- ## GrowthBook Command Line Interface (CLI) import ExternalLink from '@site/src/components/ExternalLink' The GrowthBook CLI lets you manage feature flags, experiments, metrics, and everything else in the [GrowthBook REST API](/app/api) from your terminal. Every API resource has a command group, so anything you can do via the API you can do from a shell — interactively, in scripts, or in CI. The CLI is generated from the GrowthBook OpenAPI spec, so it tracks the API closely and updates shortly after new endpoints ship. :::note Upgrading from the legacy CLI? Version 1.0 is a ground-up rewrite of the previous `growthbook` npm package. Your existing `~/.growthbook/config.toml` profiles are imported automatically on first run. Most commands are unchanged — see the [migration guide ](https://github.com/growthbook/cli/blob/main/MIGRATION.md) for what moved. ::: ## Installation Install via npm (a small launcher that downloads the prebuilt binary for your platform; requires Node.js 16+): ```bash npm install -g growthbook ``` Or with Homebrew: ```bash brew install growthbook/tap/growthbook ``` Prefer a standalone binary with no Node.js dependency? An install script, `go install`, and prebuilt downloads are covered in the [repository README ](https://github.com/growthbook/cli#cli-installation). Verify the install: ```bash growthbook --version ``` ## Authentication The CLI authenticates with a **Secret Key** or **Personal Access Token**, which you can create under [**Settings → API Keys** ](https://app.growthbook.io/settings/keys). For interactive setup, run: ```bash growthbook auth login ``` This stores your credentials in the OS keychain when available (with a config-file fallback). Check what's active at any time with: ```bash growthbook whoami ``` You can also pass credentials explicitly — useful for one-off commands or scripts: ```bash growthbook features list --bearer-auth secret_abc123 # or GBCLI_BEARER_AUTH=secret_abc123 growthbook features list ``` ### Self-hosted instances Point the CLI at your API host with `--server-url` (note the `/api` suffix), or store it once via `growthbook configure`: ```bash growthbook features list --server-url https://gb-api.example.com/api ``` ### Multiple organizations or servers Named profiles bundle a server URL and credential together: ```bash growthbook profiles set staging --server-url https://gb.staging.example.com/api --bearer-auth secret_xxx growthbook profiles use staging # set the default growthbook features list --profile prod # or per-command ``` ## Everyday usage Every command and group documents itself — start with `growthbook --help` and drill down: ```bash growthbook features list # list feature flags growthbook features get --id my-flag # inspect one growthbook features toggle --id my-flag --environments '{"production": false}' growthbook experiments list ``` For guided exploration, `growthbook explore` opens an interactive browser of the full command tree. Write operations accept individual flags or a raw JSON body (via `--body`, or piped through stdin): ```bash echo '{"id": "my-flag", "valueType": "boolean", "defaultValue": "true"}' | growthbook features create --body - ``` Feature changes that need review can go through the full draft workflow — create a revision, edit it, and publish (or request review): ```bash growthbook feature-revisions create --id my-flag --title "Enable for beta users" growthbook feature-revisions add-rule --id my-flag --version-param 2 --rule '{"type": "force", "value": "true", "condition": "{\"beta\": true}", "environments": ["production"]}' growthbook feature-revisions publish --id my-flag --version-param 2 ``` ## Scripting and automation The CLI is built to compose with shell pipelines: - `--output-format json|yaml|toon|pretty` controls the output shape (`-o json` for pipelines). - `--jq ''` filters output with built-in [jq](https://jqlang.github.io/jq/) — no separate install needed. - `--all` auto-paginates list commands, streaming NDJSON in JSON mode. - `--dry-run` prints the HTTP request that would be sent without executing it. - Exit codes are meaningful: `0` on success, non-zero on any failure, with structured error output. ```bash growthbook features list --all -o json --jq '.id' | sort ``` In CI, authenticate with the `GBCLI_BEARER_AUTH` environment variable (the keychain isn't available in headless environments) and pass `--no-interactive` to disable all prompts. ### AI coding agents When the CLI detects it's running under an AI coding agent (Claude Code, Cursor, and others), it automatically enables `--agent-mode`: structured, machine-readable errors with remediation hints and token-efficient TOON output. The CLI is self-documenting enough that agents can discover and use the full API surface from `--help` alone. Use `--agent-mode=false` to opt out. ## Generating TypeScript types Generate an `AppFeatures` type definition from your feature flags for [strictly-typed SDK usage](/lib/js#strict-typing): ```bash growthbook generate-types --output ./types ``` Add it to your `package.json` scripts to keep types in sync: ```json { "scripts": { "type-gen": "growthbook generate-types --output ./types" } } ``` ## API versioning and server compatibility REST endpoints are versioned by path prefix (`/v1`, `/v2`). Each command group targets the newest version of its endpoint; superseded versions remain available under a `-vN` suffix (e.g. `features-v1`, deprecated). When a command group advances to a newer API version, it ships as a major CLI release with a changelog callout so you can pin the prior major or switch to the explicit `-vN` command on your own schedule. **Self-hosted:** the full command surface requires **GrowthBook 4.4.0 or newer** (when the v2 API shipped). On older servers, v2-backed commands return 404 — use the `-v1` command variants or upgrade. GrowthBook Cloud is always current. The CLI checks once a day for a newer version and prints a notice to stderr when an upgrade is available _and_ compatible with the server you're connected to. Disable with `--no-update-check` or `GBCLI_NO_UPDATE_CHECK=1`. ## Detailed command reference See the generated [command documentation on GitHub ](https://github.com/growthbook/cli/tree/main/docs), or run `growthbook --help` — every command's full usage, flags, and examples are built in. --- ## Visual Studio Code Extension import ExternalLink from '@site/src/components/ExternalLink' The GrowthBook Visual Studio Code extension allows you to see your available feature definitions right in VS Code. It can be configured to work with your project in under a minute and makes it easier for developers who use VS Code to work with the GrowthBook SDK by making features browsable and conveniently surfacing common actions. You [download it from the Visual Studio Marketplace ](https://marketplace.visualstudio.com/items?itemName=GrowthBook.growthbook). --- ## Experimentation Best Practices # Experimentation Best Practices ## Running experiments ### Running Your First Experiment When you’ve finished integrating your experimentation platform (which for GrowthBook, is adding the SDK to your code), it’s time to start running an experiment. We suggest that you first run an A/A test to validate your experimentation implementation is correctly splitting traffic, and producing statistically valid results. ### Sample Sizes Understanding experiment power and MDE are important to predict how many samples are required. There are numerous online calculators that can be used to help you predict the sample size. Typical rule of thumb for the lowest number of samples required is that you want at least 100 conversion events per variation. So for example if you have a registration page which has a 10% conversion rate, and you have a 2 way (A and B) experiment that is looking to improve the member registrations, you will want to expose the experiment to at least 2,000 people (1000 per variation). ### Test Duration Due to the natural variability in traffic day to day and hour to hour, experimentation teams will often set a minimum test duration within which a test cannot be called. This helps you avoid optimizing a product for just the users that happen to visit when the test is started. For example, if the weekend traffic of your product is different from the traffic during the week, if you started a test on Friday and ended it on Monday, you may not get a complete picture of the impact your changes have to your weekday traffic. Typical test durations are 1 to 2 weeks, and usually care needs to be taken over holidays. You may also find that a test would need to run for a month or more to get the power required for the experiment. Very long running tests can be hard to justify as you have to keep the variations of the experiment unchanged for duration, and this may limit your team's velocity towards potentially higher impact changes. ### Interaction Effects and Mutual Exclusion When you start having the ability to run a lot of A/B tests, it can be tempting to not want to run tests in parallel in case they have interaction effects (see above). For example you may want to test a change in the CTA button on your purchase page, and also test changing the price. It can be difficult to figure out if any two tests will meaningfully interact, and many will run the tests in serial in an abundance of caution. However, meaningful interactions are actually quite rare, and keeping a higher rate of experimentation is usually more beneficial. You can run analysis after the experiments to see if there were any interaction effects which would change your conclusions (GrowthBook is working on an integrated solution for this). If you need to run mutually exclusive tests, you can use GrowthBook’s [namespace](/features/rules#namespaces) feature. ### Experimentation Frequency Having a high frequency of A/B testing is important for running a successful experimentation program. The main reasons why experimentation frequency is important are: **Maximizing chances**: Since success rates are typically low for any given experiment, and large changes are even more rare, by having a high frequency of A/B testing you are maximizing your chance of having impactful experiments. **Continuous improvement**: A high frequency of A/B testing allows you to continuously improve your website or application. By testing small changes frequently, you can quickly identify and implement changes that improve user experience, engagement, and conversion rates. **Adaptability**: A high frequency of A/B testing allows you to quickly adapt to changes in user behavior, market trends, or other external factors that may impact your website or application. By testing frequently, you can identify and respond to these changes more quickly, ensuring that your site or app remains relevant and effective. **Avoiding stagnation**: A high frequency of A/B testing can help you avoid stagnation and complacency. By continually testing and experimenting, you can avoid falling into a rut or becoming overly attached to a specific design or strategy, and instead remain open to new ideas and approaches. :::tip Quote **_“If you want to have good ideas you must have many ideas. Most of them will be wrong, and what you have to learn is which ones to throw away.”_** - Linus Pauling ::: --- ## Where Experimentation goes wrong # Where Experimentation goes wrong The following contains a list of common pitfalls and mistakes that can happen when running A/B tests. It is important to be aware of these issues and to take steps to avoid them in order to ensure that your A/B tests are valid and reliable. It is by no means an exhaustive list. ### Multiple Testing Problem The multiple testing problem refers to the issue that arises when statistical hypothesis testing is performed on multiple variables simultaneously, leading to an increased likelihood of incorrectly rejecting a true null hypothesis ([Type I error](/using/fundamentals#false-positives-type-i-errors-and-false-negatives-type-ii-errors)). For example, if you test the same hypothesis at a 5% level of significance for 20 different metrics, the probability of finding at least one statistically significant result by chance alone is around 64%. This probability increases as the number of tests performed increases. This math assumes that the metrics are independent from one another, which in most cases for a digital application there will be some interaction between metrics (ie, page views is most likely related to sales funnel starts, or member registration to purchase events) To address this problem, various multiple comparison correction methods can be used, such as the Bonferroni correction, False Discovery Rate (FDR) correction, or the Benjamini-Hochberg procedure. These methods adjust the significance level or the p-value threshold to account for the increased risk of false positives when multiple comparisons are made. It's essential to be aware of this issue and select an appropriate correction method when conducting multiple statistical tests to avoid false discoveries and improve the accuracy and reliability of research findings. If you are using a high number of metrics, draw conclusions from the test thoughtfully and consider running a follow-up test on that one result or metric. ### Texas Sharpshooter Fallacy The Texas sharpshooter problem is a cognitive bias that involves cherry-picking data clusters to suit a particular argument, hypothesis, or bias. The name comes from the idea of a Texan marksman shooting at a barn and then painting a target around the cluster of bullet holes to create the appearance of accuracy. As the story goes, he then showed his neighbors and convinced them he was a great shot. It is closely related to the Multiple Testing Problem/Multiple Comparison Problem. In the context of data analysis and statistics, the Texas sharpshooter problem refers to the danger of finding apparent patterns or correlations in data purely by chance and then using those patterns as if they were meaningful. This can lead to false conclusions and misguided decision-making. Texas sharpshooter problem is relevant in the sense that if you analyze the results of a test without a clear hypothesis or before setting up the experiment, you may be susceptible to finding patterns that are purely due to random variation. If you analyze the data in multiple ways or look at various subgroups without adjusting for multiple comparisons, you might identify spurious patterns that do not actually represent a true effect. ### P-Hacking P-hacking, or data dredging, is a statistical fallacy that involves manipulating or analyzing data in various ways until a statistically significant result is achieved. It occurs when researchers or analysts repeatedly test their data using different methodologies or subsets of the data until they find a statistically significant result, even if the observed effect is due to chance. In the context of A/B testing, p-hacking can be a significant concern. A/B testing involves comparing at least two versions (A and B) to determine which performs better. The danger of p-hacking arises when analysts, either consciously or unconsciously, explore different metrics, time periods, or subgroups until they find a statistically significant difference between the A and B groups. ### Peeking The peeking problem refers to the issue of experimenters making decisions about the results of an experiment based on early data. The more often the experiment is looked at, or ‘peeked’, the higher the false positive rates will be, meaning that the results are more likely to be significant by chance alone. Peeking typically applies to Frequentist statistics, which are statistically valid at their predetermined sample size. However, Bayesian statistics can also suffer from peeking depending on how decisions are made on the basis of Bayesian results. The peeking problem in A/B testing occurs when the experimenter looks at the data during the experiment and decides to stop the test early based on the observed results, rather than waiting until the predetermined sample size or duration has been reached. This can lead to inflated false positive rates, as the results are more likely to be significant by chance alone if the experimenter stops the test early based on what they see in the data. The more often the experiment is looked at, or ‘peeked’, the higher the false positive rates will be. To avoid the peeking problem in A/B testing, it's important to use a predetermined sample size or duration for the experiment and stick to the plan without making any changes based on the observed results. This helps to ensure that the statistical test is valid and that the results are not influenced by experimenter bias. Another way to avoid the peeking problem in A/B testing is to use a statistical engine that is less susceptible to peeking, like a Bayesian with custom priors, or to use a method that accounts for peeking like Sequential testing. ### Problems with client side A/B testing Client-side A/B testing is a technique where variations of a web page or application are served to users via JavaScript on the user's device, without requiring any server-side code changes. This technique can offer a fast and flexible way to test different variations of a website or application, but it can also present some potential problems, one of which is known as "flickering." Flickering is a problem that can occur when the A/B test is implemented in a way that causes the user interface to render the original version, then flash or flicker as the variations are loaded. This can happen when the A/B test code is slow to load or when the A/B testing library is lacking performance. As a result, the user may see the original version of the page briefly before it is replaced with one of the variations being tested, leading to a jarring and confusing user experience. This flickering can lead to inaccurate or unreliable test results. Rather counterintuitively, flickering may have a positive effect on the results, sometimes the flashing may draw a users attention to that variation, and cause an inflation in the effect. To avoid flickering in client-side A/B testing, it is important to implement the test code in a way that minimizes the delay between the original page and the variations being tested. This may involve preloading the test code or optimizing the code for faster loading times. GrowthBook’s SDKs are built for very high performance, and allow you to use client side A/B testing code inline, so there are no blocking 3rd party calls. You can also use an alternative technique such as server-side testing or redirect-based testing to avoid flickering issues. If loading the SDK in the head does not sufficiently prevent flicking, you can also use an anti-flickering script. These scripts hide the page while the content is loading, and reveal the page after the experiment loaded. The problem with this is that while it technically prevents flickering, it slows how quickly your site appears to load. ### Redirect tests (Split testing) Redirect-based A/B testing is a technique where users are redirected to different URLs or pages based on the A/B test variation they are assigned to. While this technique can be effective in certain scenarios, it can also present several potential problems that should be considered before implementation. **SEO**: Redirects can negatively impact SEO, as search engines may not be able to crawl the redirected pages or may see them as duplicate content. This can result in lower search engine rankings and decreased traffic to the site. **Load times/User experience**: Redirects can increase page load times, as the browser has to make an additional HTTP request to load the redirect page. This can result in slower load times, which can impact user experience, conversion rates, and A/B test outcomes. **Data accuracy**: Redirects can also impact the accuracy of the test results, as users may drop off or exit the site before completing the desired action due to a slower load time or confusing user experience. It can also be harder technically to fire the tracking event, causing a loss in data. To mitigate these problems, it's important to carefully consider whether redirect-based A/B testing is the most appropriate technique for your specific use case. If you do choose to use redirects, it's important to implement them correctly and thoroughly test them to ensure that they do not negatively impact user experience or test results. Additionally, it may be helpful to use other techniques such as server-side testing or client-side testing to supplement redirect-based testing and ensure the accuracy and reliability of the test results like testing on the edge or using middleware to serve different pages. ### Semmelweis Effect The Semmelweis effect refers to the tendency of people to reject new evidence or information that challenges their established beliefs or practices. It is named after Ignaz Semmelweis, a Hungarian physician who, in the 19th century, discovered that hand washing could prevent the spread of infectious diseases in hospitals. Despite his findings, he was ridiculed and ignored by his colleagues, and it took many years for his ideas to be accepted and implemented. In the context of A/B testing, the Semmelweis effect can manifest in several ways. For example, a company may have a long-standing belief that a certain design or feature is effective and produces good results, and may not want to experiment with it because everyone knows it ‘correct’. Even if an experiment is run against this entrenched belief, and the results of an A/B test challenge established norms, there may be resistance to accept the new evidence and change the established practice. To avoid the Semmelweis effect in A/B testing, it is important to approach experimentation with an open mind and a willingness to challenge established beliefs and practices. It is crucial to let the data guide decision-making and to be open to trying new things, even if they go against conventional wisdom or past practices. It is also important to regularly review and evaluate the results of A/B tests to ensure that the company's beliefs and practices are aligned with the latest evidence and insights, and haven’t changed over time. ### Confirmation Bias Confirmation bias refers to the tendency to favor information that confirms our preexisting beliefs and to ignore or discount information that contradicts our beliefs. In the context of A/B testing, confirmation bias can lead to flawed decision-making and missed opportunities for optimization. For example, if a company believes that a certain website design or feature is effective, they may only run A/B tests that confirm their beliefs and ignore tests that challenge their beliefs. This can lead to a bias towards interpreting data in a way that supports preexisting beliefs, rather than objectively evaluating the results of the tests. Or a PM may believe a new version of their product will be superior, and only acknowledge evidence that confirms this belief. Confirmation bias can also manifest in the way tests are designed and implemented. If a company designs an A/B test in a way that biases the results towards a particular outcome, such as by using a biased sample or by selecting a suboptimal metric to measure success, it can lead to misleading results that confirm preexisting beliefs. To avoid confirmation bias in A/B testing, it is important to approach experimentation with an open and objective mindset. This involves being willing to challenge preexisting beliefs (Semmelweis) and being open to the possibility that the data may contradict those beliefs. It also involves designing tests in a way that is unbiased and that measures the most relevant and meaningful metrics to evaluate success. Having multiple stakeholders review and evaluate the results of A/B tests can help ensure that decisions are based on objective data, rather than personal biases or beliefs. ### HiPPOs HiPPO is an acronym for the "highest paid person's opinion." In less data-driven companies, decisions about what product to build or which products to ship are made by HiPPOs. The problem with HiPPOs is that it turns out their opinions are no more likely to be right than anyone else's opinions, and are therefore often wrong. But due to their status they may resist against experimentation to preserve their status or ego. The HiPPO effect is a common problem in many organizations, and it can lead to poor decision-making and missed opportunities for your product. ### Trustworthiness When experiment results challenge existing norms or an individual’s beliefs, it can be easy to blame the data. For this reason, having a trustworthy A/B testing platform is extremely important. There must be ways to audit the results, and look into if there was any systemic or specific problem affecting the results of the experiment. Running A/A tests can help build trust that the platform is working correctly. Trust in an experimentation platform is built over time, and care must be taken to not just dismiss results that are counterintuitive. ### Twyman's Law Twyman's law is a principle in statistics that states that any data that is measured and collected will contain some degree of error, and that this error is an inherent part of the data. It is named after the British statistician Maurice G. Kendall Twyman. In the context of A/B testing, Twyman's law suggests that there will always be some level of variability or uncertainty in the results of an A/B test due to factors such as random chance, sample size, or measurement error. It is often phrased as: > Any data or figure that looks interesting or different is usually wrong If you notice a particularly large or unusual change in the results of an experiment, it is more likely to be the result of a problem with the data or an implementation than an actual result. Before you share the results, make sure that the effects are not the result of an error. ### Goodhart's Law Goodhart's law is a concept in economics that states that when a measure becomes a target, it ceases to be a good measure. In other words, once a metric becomes the sole focus of attention and effort, it loses its value as an indicator of the desired outcome. When it comes to A/B testing, Goodhart's law can apply in several ways. For example, if a specific metric such as click-through rate or conversion rate becomes the sole focus of an A/B test, it can lead to unintended consequences such as artificially inflating the metric while neglecting other important aspects of the user experience. This can happen because individuals or teams may optimize for the metric being measured rather than focusing on the broader goals of the A/B test, such as improving user engagement or increasing revenue. To avoid the negative effects of Goodhart's law in A/B testing, it is important to choose the right metrics to track and analyze, and to use a variety of metrics to evaluate the effectiveness of the test. It is also important to keep in mind the broader goals of the test and to avoid tunnel vision on any one metric. Goodhart's law is more likely to happen when you are using proxy metrics, instead of the real KPIs you’re trying to improve - an example of this might be items added to a cart being used as a proxy for purchases. Also If the proxy metric is not strongly causally linked to the target metric, pressing hard on the proxy may have no effect on the goal metric, or might actually cause the correlation to break. ### Simpson's Paradox Simpson's paradox is a statistical phenomenon where a trend or pattern appears in different groups of data but disappears or reverses when the groups are combined. In other words, the overall result may be opposite to what the individual subgroups suggest. This paradox can arise when a confounding variable (a variable that affects both the independent and dependent variables) is not taken into account while analyzing the data. Simpson's paradox was famously observed at the University of California, Berkeley in 1973, where it had implications for gender discrimination in graduate school admissions. At the time, it was observed that although the overall admission rate for graduate school was higher for men than for women (44% vs. 35%), when the admission rates were broken down by department, the reverse was true for many of the departments, with women having a higher admission rate than men in each department. In the Department of Education, for example, women had a 77% admission rate compared to men's 62% admission rate. The paradox was resolved by examining the application data more closely and considering the impact of an important confounding variable, which was the choice of department. It was discovered that women were more likely to apply to departments that were more competitive and had lower admission rates, while men were more likely to apply to less competitive departments with higher admission rates. When the data was reanalyzed, taking into account the departmental differences in admission rates, it was found that women actually had a slightly higher overall admission rate than men, suggesting that there was no discrimination against women in the admissions process. This case study illustrates how Simpson's paradox can occur due to the influence of confounding variables, and how it can lead to misleading conclusions if not properly accounted for in the analysis. To avoid the Simpson's paradox in experimentation, it is essential to analyze the data by considering all relevant variables and subgroups. It is crucial to ensure that the experimental groups are similar in terms of demographics and behavior, and to use statistical techniques that account for confounding variables. ### Ethical considerations Experimentation judges the outcome of changes by looking at the impact it has on some set of metrics. But the seeming objectivity of the results can hide problems. The simplest way this can go wrong is if your metrics are tracking the wrong things, in which case you'll have garbage in and garbage out. Furthermore, relying solely on short-term metrics can encourage [dark patterns](https://blog.growthbook.io/dark-patterns-a-b-testing/) in A/B testing, where you inadvertently exploit user trust to boost numbers temporarily at the expense of long-term retention. But it is also possible for the metrics to not capture harm that is being done to some subsets of your population. Experimentation results work on averages, and this can hide a lot of systemic biases that may exist. There can be a tendency for algorithmic systems to “learn” or otherwise encode real-world biases in their operation, and then further amplify/reinforce those biases. Product design has the potential to differentially benefit some groups of users more than others; It is possible to measure this effect and ensure that results account for these groups. Sparse or poor data quality that leads to objective-setting errors and system designs that lead to suboptimal outcomes for many groups of end users. One company that does this very well is the team at LinkedIn, you can read about their approach [here](https://engineering.linkedin.com/blog/2020/building-inclusive-products-through-a-b-testing). --- ## Experimenting in GrowthBook import MaxWidthImage from "@site/src/components/MaxWidthImage"; # Experimenting in GrowthBook The Experiments section in GrowthBook is all about analyzing raw experiment results in a data source. Before analyzing results, you need to actually run the experiment. This can be done in several ways: - Feature Flags (most common) - Running an inline experiment directly with our SDKs - Our [Visual Editor](/app/visual) - Your own custom variation assignment / bucketing system When you go to add an experiment in GrowthBook, it will first look in your data source for any new experiment ids and prompt you to import them. If none are found, you can enter the experiment settings yourself. ## Experiment Splits When you run an experiment, you need to choose who will get the experiment, and what percentage those users should get each variation. In GrowthBook, we allow you to pick overall exposure percentage, as well as customize the split per variation. Yor can also target an experiment at just some attribute values. GrowthBook uses deterministic hashing to do the assignment. That means that each user’s hashing attribute (usually user id), and the experiment name, are hashed together to get a number from 0 to 1. This number will always be the same for the same set of inputs. There is quite often a need to de-risk a new A/B test by running the control at a higher percentage of users than the new variation, for example, 80% of users get the control, and 20% get the new variation. To solve this case, we recommend keeping the experiment spits equal, and adjusting the overall exposure (ie, 20% exposure, 50/50 on each variation, so each variation gets 10%). This way the overall exposure can be ramped up (or down) without having any users potentially switch variations. ## Metric selection GrowthBook lets you choose goal metrics, secondary metrics, and guardrail metrics. Goal metrics are the metrics you’re trying to improve or measure the impact of the change of your experiment. Secondary metrics are used to learn about experiment impacts, but are not primary objectives. Guardrail metrics are metrics you’re trying not to hurt. ![GrowthBook Metric Selector](/images/using/metrics-modal.png) It is best to pick metrics for your experiment that are as close to your treatment as possible, and, if possible, the event itself. For example, if you’re trying to improve a signup rate, you can add product review metrics that are close to that event, like "signup modal open rate", and "signup conversion rate". You can add as many metrics as you like to your experiment, but we suggest each experiment have only a few primary metrics that are used for making the shipping decision. Adding all your metrics is not recommended, as this can lead to false positives caused by random variations (see [Multiple testing problem](/using/experimentation-problems#multiple-testing-problem)) Before running a test, select a primary metric or set of metrics that you're trying to improve. These metrics are often called the OEC for Overall Evaluation Criterion. It is important to have this decided ahead of time so when you look at your experiment results you're not just shopping for metrics that confirm your bias (see [confirmation bias](/using/experimentation-problems#confirmation-bias)). With GrowthBook, Goal and guardrail metrics can be added retroactively to experiments, as long as the data exists in your data warehouse. This allows you to reprocess old experiments if you add new metrics or redefine a metric. ## Activation metrics Assigning your audience to the experiment should happen as close to the test as possible to reduce noise and increase power. However, there are times when running an experiment requires that users be bucketed into variations before knowing that they are actually being exposed to the variation. One common example of this is with website modals, where the modal code needs to be loaded on the page with the experiment variations, but you’re not sure if each user will actually see the modal. With activation metrics you can specify a metric that needs to be present to filter the list of exposed users to just those with that event. ## Sample sizes and metric totals When running an experiment you select your goal metrics. Getting enough samples depends on the size of the effect you’re trying to detect. If you think the experiment will have a large effect, the smaller total number of events you need to collect. GrowthBook allows users to set a minimum metric total for each metric where we will hide results before that threshold is reached to avoid premature peeking. ## Test Duration We recommend running an experiment for at least 1 or 2 weeks to capture variations in your traffic. Before a test is significant, GrowthBook will give you an estimated time remaining before it reaches the minimum thresholds. Traffic to your product is likely not uniform, and there may be differences ## Metric Windows A lot can happen between when a user is exposed to an experiment, and when a metric event is triggered. How you want to attribute that conversion event to the experiment is adjustable within GrowthBook using metric and experiment level settings. At the metric level, you can pick three different metric windows: - None - uses as much data as possible from the user's exposure until the end of the experiment. - Conversion - uses only data in some window after a user's first exposure. If the metric's conversion window is set to 72 hours, any conversion that happens after that is ignored. - Lookback - uses only data in the last window before an experiment ends. Here's a representation of how these metric windows work for a hypothetical user: ![Metric Windows](/images/metric-windows.png) Here's a second example for a hypothetical User 2, who joins the experiment late. Notice that the conversion window can extend beyond the experiment end date. ![Metric Windows (User 2)](/images/metric-windows-user-2.png) You can override all Conversion windows to be No window at the experiment level using the "Conversion Window Override" in the Experiment Analysis Settings. ## Understanding results ### Bayesian Results In GrowthBook the experiment results will look like this. ![GrowthBook Results](/images/using/experiment-results-bayesian.png) Each row of this table is a different metric. This is a simplified overview of the data. If you want to see the full data, mouse over any of the results. ![GrowthBook Results](/images/using/experiment-results-bayesian-details2.png) Value is the conversion rate or average value per user. In small print you can see the raw numbers used to calculate this. **Chance to Beat Control** tells you the probability that the variation is better. If you are familiar with Frequentist statistics, you can consider this value 1 - the p value. Anything above the threshold (which by default is set to 95%) is highlighted green indicating a very clear winner. Anything below the threshold (5% by default) is highlighted red, indicating a very clear loser. Anything in between is gray indicating it's inconclusive. If that's the case, there's either no measurable difference or you haven't gathered enough data yet. **Percent Change** shows how much better/worse the variation is compared to the control. It is a probability density graph and the thicker the area, the more likely the true percent change will be there. As you collect more data, the tails of the graphs will shorten, indicating more certainty around the estimates. ### Frequentist Results You can also choose to analyze results using a Frequentist engine that conducts simple t-tests for differences in means and displays the commensurate p-values and confidence intervals. If you selected the "Frequentist" engine, when you navigate to the results tab to view and update the results, you will see the following results: ![GrowthBook Results - Frequentist](/images/using/experiment-results-frequentist.png) Everything is the same as above except for two key changes: - The Chance to Beat Control column has been replaced with the P-value column. The p-value is the probability that the percent change for a variant would have been observed if the true percent change were zero. When the p-value is less than the threshold (default to 0.05) and the percent change is in the preferred direction, we highlight the cell green, indicating it is a clear winner. When the p-value is less than the threshold and the percent change is opposite the preferred direction, we highlight the cell red, indicating the variant is a clear loser on this metric. - We now present a 95% confidence interval rather than a posterior probability density plot. ## Data quality checks GrowthBook performs automatic data quality checks to ensure the statistical inferences are valid and ready for interpretation. You can check and monitor the health of your experiments on the experiment **health page**. ### Health Page GrowthBook automatically does data quality checks on all experiments and shows the results on the **Health Page**. ![Experiment Health Page](/images/using/health-page.png) This page shows experiment exposure over time, and also all the other health checks we do. ### Little traffic in experiment Experiment Traffic Over Time shows how many users are in the experiment at a given timepoint. If traffic is too low, please see our [troubleshooting guide](/kb/experiments/troubleshooting-experiments#problem-1-no-or-little-traffic-flowing-into-the-experiment). ### Sample Ratio Mismatch (SRM) Every experiment automatically checks for a Sample Ratio Mismatch and will warn you if found. This happens when you expect a certain traffic split (e.g. 50/50) but you see something significantly different (e.g. 46/54). We only show this warning if the p-value is less than 0.001, which means it's extremely unlikely to occur by chance. We will show this warning on the results page, and also on our experiment health page. ![Sample Ratio Mismatch](/images/using/srm-check-health-page.png) Like the warning says, you shouldn't trust the results since they are likely misleading. Instead, find and fix the source of the bug and restart the experiment. You can find more information about potential sources of the problems in our [troubleshooting guide](/kb/experiments/troubleshooting-experiments#problem-2-srm-errors-traffic-imbalance-in-the-experiment). ### Multiple Exposures We also automatically check each experiment to make sure that too many users have not been exposed to multiple variations of a single experiment. This can happen if the hashing attribute is different from the assignment id used in the report, or for implementation problems. Please see our [troubleshooting guide](/kb/experiments/troubleshooting-experiments#problem-3-multiple-exposures). ### Minimum Data Thresholds You can set thresholds per metric to make sure people viewing the results aren’t drawing conclusions too early (e.g. when it’s 5 vs 2 conversions) ### Variation Id Mismatch GrowthBook can detect missing or improperly-tagged rows in your data warehouse. The most common way this can happen if you assign with one parameter, but send a different ID to your warehouse from the trackingCallback call. It may indicate that your variation assignment tracking is not working properly. ### Suspicious Uplift Detection You can set thresholds per metric for a maximum percent change. When a metric results is above this, GrowthBook will show an alert. Large uplifts may indicate a bug - see [Twymans Law](/using/experimentation-problems#twymans-law). ### Guardrails Guardrail metrics are ones that you want to keep an eye on, but aren't trying to specifically improve with your experiment. For example, if you are trying to improve page load times, you may add revenue as a guardrail since you don't want to inadvertently harm it. Guardrail results show up beneath the main table of goal and secondary metrics. The full statistics are shown like goal metrics, and similarly they are colored based on "Chance to Beat Control". If guardrail metrics become significant, you may want to consider ending the experiment. ![Guardrail Results](/images/using/guardrail-metrics.png) If you select the frequentist engine, we instead use yellow to represent a metric moving in the wrong direction at all (regardless of statistical significance), red to represent a metric moving in the wrong direction with a two-sided t-test p-value below 0.05, and green to represent a metric moving in the right direction with a p-value below 0.05. Otherwise the cell is unshaded if the metric is moving in the right direction but not statistically significant at the 0.05 level. ## Digging deeper GrowthBook lets you dig into the results to get a better understanding of the likely effect of your change. ### Segmentation :::caution Segments are deprecated Segments have been deprecated and are being phased out. If you don't already have one configured, you shouldn't create one. Use a [Dimension](/app/dimensions) instead, or apply a custom SQL filter in the experiment's analysis settings to only show users that match a particular attribute. ::: Previously, Segments could be applied to experiment results to only show users that match a particular attribute. For example, you might have “country” as a dimension, and create a segment for just “US visitors”. In the experiment you could configure the experiment to just look at one particular segment of users. If you need this kind of filtering today, a custom SQL filter in the experiment's analysis settings is the recommended approach, as long as the attribute you want to filter on already exists on your experiment assignment query. ### Dimensions GrowthBook lets you break down results by any dimension you know about your users. We automatically let you break down by date, and any additional dimensions can be added either with the exposure query, or with custom SQL from the dimension menu. Some examples of common dimensions are “Browser” or “location”. You can read more about [dimensions here](/app/dimensions). ![Dimension Selector](/images/using/dimension-selector.png) It can be very helpful to look into how specific dimensions of your users are affected by the experiment. For example, you may discover that a specific browser is underperforming compared to the rest, and this may indicate a bug, or something to investigate further. The more metrics and dimensions you look at, the more likely you are to see a false positive. If you find something that looks surprising, it's often worth a dedicated follow-up experiment to verify that it's real. ### Custom reports Custom reports create snapshots of experiment results, allowing you to capture and share specific analyses or points in time. Common use cases for custom reports include: - Capturing milestone results at specific points in the experiment - Analyzing specific date ranges or segments - Adding or removing metrics for focused analysis - Applying custom SQL filters to handle outliers - Creating targeted views for different stakeholders - Documenting key findings for future reference ![Example Custom Reports](/images/reports-share-report.webp) For step-by-step instructions on creating and managing custom reports, see our [Sharing Experiment Insights](/app/experiment-results#sharing-experiment-insights) documentation. ## Deciding A/B test results Hopefully you are analysing your experiment results with your OEC already documented. Even so, when to stop, and how to interpret results may not be straight forward. ### When to stop an experiment When using the Bayesian statistics engine, there are a few methods you can use when stopping a test. - significance reached on your primary metrics - confidence intervals do not include harmful outcomes - guardrail metrics are not affected - test duration reached It all depends on what you’re trying to do with the experiment. For example, if you’d like to know what impact your change has, you should use the first method. If you’re doing a design change, and want to make sure you haven’t broken anything on your product, you can use the guard rail approach. You should also make sure that the experiment has run for your minimum test duration (typically 1 or 2 weeks), so that you’re not looking at highly skewed sampling. For Frequentist statistics, you should determine the running time of the experiment and stop the test at that fixed horizon to ensure accurate results (see [Peeking](/using/experimentation-problems#peeking)) or use Sequential analysis. ### Interpreting results It is quite common to have experiment results with mixed results. Deciding on the results of an experiment in these cases may require some interpretation. As a general rule, you should have one goal metric that is the primary metric you’re trying to improve, and if this metric is up significantly it is generally straightforward to declare a result. If you have a mix and up and down metrics, the decisions are less clear. Once you have reached a decision with your experiment, you can click the “mark as finished” link towards the top of the results. This will open a modal where you can document the results, including the result, and observations. This creates a card on the top of the experiment results with your conclusion. Please note that currently this marking a test as finished does not stop the test from running. If you are using feature flags to run the experiment, you should also go to the feature and turn off the experiment. ### Inconclusive results Sometimes you may have an experiment that is inconclusive. Generally it is a good idea to have a policy of what to do in these cases. We suggest that your policy should be to revert to the control variant in these cases, unless the new version unlocks some new features. --- ## A/B Testing Fundamentals # A/B Testing Fundamentals If you are new to A/B testing, you may find a lot of new terminology. The goal of this section is to help you understand the basics of A/B testing. ## Glossary - Common Experimentation Terms ### Control (or Baseline) The existing version of the product that you are trying to improve upon. ### Variation (or Treatment) A new version of the page that you are testing against the Control. ### Hypothesis Formal way to describe what you are changing and what you think it will do. ### Statistical Significance An indicator that the difference in performance between the control and treatment groups is unlikely to have occurred by chance. ### Confidence level The level of certainty we want before the result of a test is statistically significant. A common confidence level used in A/B testing is 95%. ### Sample size The number of visitors or users who are included in the A/B test. ### Test duration The length of time that the A/B test is run. This can vary depending on the sample size and the desired confidence level. ### Variance The degree to which the results of an A/B test vary over time or across different segments of the user base. ## Anatomy of an A/B test ![Anatomy of an A/B test](/images/using/ab-test-diagram.png) Hypothesis Come up with an idea you want to test Assignment Randomly split your audience into persistent groups Variations Create and show different experiences to each group Tracking Record events and behaviors of the two groups Results Use statistics to determine if the differences in behavior are significant ### Hypothesis Good A/B tests, and really any project, starts with a hypothesis about what you’re trying to do. A good hypothesis should be as simple, specific, and falsifiable as possible. A good A/B test hypothesis should be: - **Specific**: The hypothesis should clearly state what you want to test and what outcome you expect to see. - **Measurable**: The hypothesis should include a metric or metrics that can be used to evaluate the outcome of the test. - **Relevant**: The hypothesis should be relevant to your business goals and objectives. - **Clear**: The hypothesis should be easy to understand and communicate to others. - **Simple**: The fewer variables that are involved in the experiment, the more causality can be implied in the results. - **Falsifiable**: The hypothesis should be something that can be tested using an A/B test to determine the validity of the hypothesis. Overall, a good A/B test hypothesis should be a clear statement that identifies a specific change you want to make and the expected impact on a measurable outcome, while being grounded in data and relevant to your business goals. ### Audience and Assignments Choose the audience for your experiment. To increase the detectable effect of your experiment, the audience you choose should be as close to the experiment as possible. For example, if you’re focusing on a new user registration form, you should select as your audience just unregistered users. If you were to include all users, you would have users who could not see the experiment, which would increase the noise and reduce the ability to detect an effect. Once you have selected your audience, you will randomize users to one variation or another. ### Variations An A/B test can include as many variations as you like. Typically the A variation is the control variation. The variations can have as many changes as you like, but the more you change the less certain you can be what caused the change. ### Tracking Tracking is the process of recording events and behaviors that your users do. In the context of AB testing, you want to track events that happen after exposure to the experiment, as these events will be used to determine if there is a change in performance due to being exposed to the experiment. AB testing systems either are "warehouse native" (like GrowthBook) as in they use your existing event trackers (like GA, Segment, Rudderstack, etc), or they require you to send event data to them. ### Results With A/B testing we use statistics to determine if the effect we measure on a metric of interest is significantly different across variations. The results of an A/B test on a particular metric can have three possible outcomes: win, loss, or inconclusive. With GrowthBook we offer two different statistical approaches, Frequentist and Bayesian. By default, GrowthBook uses Bayesian statistics. Each method has their pros and cons, but both will provide you with evidence as to how each variation affected your metrics. ## Experimentation Basics ### Typical success rates A/B testing can be incredibly humbling—one quickly learns how often our intuition about what will be successful with our users is incorrect. Industry wide average success rates are only about 33%. ⅓ of the time our experiments are successful in improving the metrics we intended to improve, ⅓ of the time we have no effect, and ⅓ of the time we hurt those metrics. Furthermore, the more optimized your product is, the lower your success rates tend to be. But A/B testing is not only humbling, it can dramatically improve decision making. Rather than thinking we only win 33% of the time, the above statistics really show that A/B tests help us make a clearly right decision about 66% of the time. Of course, shipping a product that won (33% of the time) is a win, but so is not shipping a product that lost (another 33% of the time). Failing fast through experimentation is success in terms of loss avoidance, as you are not shipping products that are hurting your metrics of interest. ### Experiment power With A/B testing, power analysis refers to whether a test can reliably detect an effect. Specifically, it is often written as the percent of the time a test would detect an effect of a given size with a given number of users. You can also think of the power of a test with respect to the sample size. For example: "How many times do I need to toss a coin to conclude it is rigged by a certain amount?" ### Minimal Detectable Effect (MDE) Minimal Detectable Effect is the minimum difference in performance between the control and treatment groups that can be detected by the A/B test, given a certain statistical significance threshold and power. The MDE is an important consideration when designing an A/B test because if the expected effect size is smaller than the MDE, then the test may not be able to detect a significant difference between the groups, even if one exists. Therefore, it is useful to calculate the MDE based on the desired level of statistical significance, power, and sample size, and ensure that the expected effect size is larger than the MDE in order to ensure that the A/B test is able to accurately detect the difference between the control and treatment groups. ### False Positives (Type I Errors) and False Negatives (Type II Errors) When making decisions about an experiment, we can say that we made the right decision when choosing to ship a winning variation or shut down a losing variation. However, because there is always uncertainty in the world and we rely on statistics, sometimes we make mistakes. Generally, there are two kinds of errors we can make: Type I and Type II errors. **Type I Errors**: also known as False Positives, these are errors we make when we think the experiment provides us with a clear winner or a clear loser, but in reality the data are not clear enough to make this decision. For example, your metrics all appear to be winners, but in reality the experiment has no effect. **Type II Errors**: also known as False Negatives, these are errors we make when the data appear inconclusive, but in reality there is a winner or a loser. For example, you run an experiment for as long as you planned to, and the data aren’t showing a clear winner or loser when actually a variation is much better or worse. Type II errors often require you to collect more data or choose blindly rather than provide you with the correct, clear answer Actual Results Inconclusive Lost Won Decision Made Inconclusive Correct Inference Type II error (false negative) Type II error (false negative) Shut down Type I error (false positive) Correct Inference Type I error (false positive) Ship Type I error (false positive) Type I error (false positive) Correct Inference ### P-Value In frequentist statistics, a p-value is a measure of the evidence against a null hypothesis. The null hypothesis is the hypothesis that there is no significant difference between two groups, or no relationship between two variables. In the context of A/B testing, the p-value is a statistical measure that indicates whether there is a significant difference between two groups, A and B. The p-value is the probability of observing a difference as extreme or more extreme as your actual difference, given there is actually no difference between groups. If the p-value is less than a predetermined level of significance (often 0.05), the result is deemed to be statistically significant as the difference is not likely due to chance. For example, let's say that you conduct an A/B test in which you randomly assign users to either group A (the control group) or group B (the experimental group). You measure a specific metric such as conversion rate for each group, and you calculate the p-value to test the hypothesis that there is no difference between the two groups. If the p-value is less than 0.05, the observed difference conversion rate between the two groups is unlikely if there wasn't truly a difference in groups; we say the effect is statistically significant and likely not due to chance. It's important to note that p-value alone cannot determine the importance or practical significance of the findings. Additionally, it's essential to consider other factors such as effect size, sample size, and study design when interpreting the results. ### A/A Tests A/A testing is a form of A/B testing in which instead of serving two different variations, two identical versions of a product or design are tested against each other. In A/A testing, the purpose is not to compare the performance of the two versions, but rather to check the consistency of the testing platform and methodology. The idea behind A/A testing is that if the two identical versions of the product or design produce significantly different results, then there may be an issue with the testing platform or methodology that is causing the inconsistency. By running an A/A test, you can identify and address any potential issues before running an A/B test, which can help ensure that the results of the A/B test are reliable and meaningful. A/A testing is a useful tool for ensuring the accuracy and reliability of A/B tests, and can help improve the trust in the platform, and faith in the quality of the insights and decisions that are based on the results of these tests. [Read more about running A/A tests in GrowthBook](/kb/experiments/aa-tests). ### Interaction effects When you run multiple tests simultaneously, there's a chance they may interfere with each other. For example, two tests that change prices in different parts of your product could cause some users to see conflicting prices, eroding trust. That's an extreme case. A more common scenario is when someone sees one experiment on the account registration page and another on the checkout page. If the tests run in parallel, users will see all combinations of variations: AA, AB, BA, and BB. A meaningful interaction effect would occur if, say, the AA combination outperforms the others by more than each test's individual effect would predict. In practice, meaningful interaction effects are rare. More often, running tests in parallel simply increases variance without changing the overall results. ### Novelty and Primacy Effects Novelty and primacy effects are psychological phenomena that can influence the results of A/B testing. The novelty effect refers to the tendency of people to react positively to something new and different. In the context of A/B testing, a new design or feature may initially perform better than an existing design simply because it is new and novel. However, over time, the novelty effect may wear off and the performance of the new design may decrease. The primacy effect refers to the tendency of people to remember and give more weight to information that they encounter first. With A/B testing, this can manifest as an initial reduction in the improvement for metrics as users prefer the original treatment of the product. One way to mitigate the effects of novelty is to run tests over a longer period of time to allow for the novelty effect to wear off. Another approach is to stagger the rollout of a new design or feature to gradually introduce it to users and avoid a sudden and overwhelming change. To account for the primacy effect, you can target or segment an experiment to just new users to ensure that they won’t be influenced by how things used to work. This can help ensure that the results of the test are truly reflective of user behavior and preferences, rather than the order in which designs were presented. --- ## GrowthBook Best Practices # GrowthBook Best Practices ## Organization As you scale up your usage of GrowthBook and start running many experiments, keeping everything organized and easy to find is essential. GrowthBook includes a number of organizational structures to help you scale. ### Organizations The organization is the highest level structure within GrowthBook. An organization contains everything within your GrowthBook instance: users, data sources, metrics, features, etc. For both cloud and self-hosted users, it is possible for users to join multiple organizations. Users can belong to multiple organizations, but each organization is otherwise entirely independent of the others. For some, complete isolation of the teams or subdivisions within the company may be desired. For example, if your company has two or more largely independent products (e.g., Google has Search and Google Docs), you can set up multiple organizations per product. For self-hosted enterprise users, we support multi-organization mode, which also comes with a super-admin account type that can manage users across organizations. ### Environments In GrowthBook, you can create as many environments as you need for feature flags and their rules. Environments are meant to separate how your feature flags and rules are deployed. Each environment can have one or more SDK API endpoints specified when you create the SDK, allowing you to differentiate the rules by environment. For example, you might have environments for “Staging”, “QA”, and “Production”. While testing the feature, you can set specific rules to on the "development" or "QA" environment, and when you're ready, you can move applicable rules to the "production" environment. You can add an arbitrary number of environments from the SDK Connections → Environments page. ![Environments Page](/images/using/environments-page.png) ### Projects Within an organization, you can create projects. Projects can help isolate the view of GrowthBook to just the sections that apply for that GrowthBook user. Projects are a great way to organizationally separate features, metrics, experiments, and even data sources by team or product feature. For example, you could have a project “front-end” and one for “back-end”, or by team like “Growth” and “API”. Unlike separate organizations, projects can share data. Projects are managed from the Settings → Projects page. ![Projects Page](/images/using/projects-page.png) A use case for using projects is if you have divisions within your product but a centralized data source. We typically see projects used per team or per project within your organization. For example, if you have a mobile app and a website that shares users, but the code bases are different, you will want to create two projects: a _mobile_ project and a _web_ project. Each of the items within GrowthBook can be assigned to multiple projects. You can have a data source that is part of the ‘mobile’ and ‘web’ projects but not to a ‘marketing’ project. That data source will not be available for users in the 'marketing' project. To help keep feature payloads smaller, the SDK endpoint where the feature definitions are returned can be scoped to each project. If using Projects based on features or area of your product, you can use this feature to only return features that pertain to that area. For example, with our “mobile” and “website” example, you can add the project scope to only return features for the project as these are likely to use different code than the other, and you don’t want to expose features unnecessarily. One advantage of using projects is that you can adjust permissions and even some statistical settings per project- users can have no access to a project or, inversely, have no general permissions but add a project permission so they can work within their project. If a team prefers to use a frequentist statistical model, this can be adjusted per project. ### Tags Another way to organize GrowthBook is with _tags_. With tags, you can quickly filter lists, and select metrics. For example, if you tagged all experiments to do with your checkout flow with the tag “checkout”, you can quickly see this in the list by clicking on ‘filter by tags’ on the experiment list. Tags can be color-coded and managed from our Settings → Tags page. You can add multiple tags per item you are tagging. ![Tags Page](/images/using/tags-page.png) Metrics with tags can be used to quickly add all those metrics to an experiment. When creating an experiment or editing the metrics, there is a section titled “Select metric by tag” which will let you add all the metrics by the tag name to guardrail, secondary, and/or goal metrics. This is useful if you want to use a standard set of metrics for your experiments. Tags are often used to mark sub-features of your product; for example, if you have an e-commerce website, you might want to tag features or experiments with the area they affect, like ‘_pricing_,’ ‘_product page_,’ or ‘_checkout_.’ ![Experiments filtered by tag](/images/using/experiments-filtered-by-tag.png) ### Naming Another organizing principle you can use is the naming of your experiments and features. Because GrowthBook makes it easy to quickly search the list of features and flags, using naming conventions can be an effective way to organize your project. We’ve seen several strategies be successful here, but as a general rule, you’ll want to be as specific as possible with naming features and experiments. For example, you can use <project scope>\_<project name> or the year, quarter, or section plus the name of the experiment, e.g.: “23-Q4 New user registration modal“ or “23-Team3 Simplified checkout flow”. This lets you quickly see when the experiment was run or which team worked on it. ### Hygiene & Archiving As the number of features and experiments grows, you will want to remove past items that are no longer relevant. Within GrowthBook you can archive and delete. **Deleting** something will permanently remove items from GrowthBook. **Archived** items in GrowthBook won’t be deleted, but they are removed from the main part of the UI and not available for adding to new experiments (for archived metrics). Archived items can also be restored at any time. These methods help you keep your UI clean and relevant. ### Source of Truth If you run an experimentation program for a long enough time, you’ll find yourself with an experiment idea that seems really familiar, and people will wonder, “Didn’t we already test this?” If you don’t have a central repository for all your experiment results, it can be difficult to find if you did test this previously, and even if you did, if what you tested was similar enough to the new idea not to have to test it again. GrowthBook is designed to help with this by creating a central source for the features you’ve launched and the experiments you’ve run. To help facilitate this, GrowthBook has created a number of features to help you capture meta information. ### Meta Information Features and experiments can all have metadata attached to them. The purpose of this is to help capture all the meta-information around a feature or experiment that might help contextualize it for posterity and help capture the institutional knowledge that your program generates. This is also very helpful when new members join your team, so they don’t just suggest ideas you’ve run many times already. By default, GrowthBook has fields for a description, hypothesis, and tags. The description is a free-form text field, and meant to capture the broader context of the feature or experiment. The hypothesis is a structured field that is meant to capture the specific hypothesis you are testing with the experiment. The tags are a way to categorize the feature or experiment. You can add multiple tags to each item. For experiments, you should capture the original idea, any screenshots of similar products, and, most importantly, capture images/screenshots of the control and variants for the experiment. Quite often, someone will suggest an idea you’ve run previously. In these cases, it is vital to be able to find out what exactly you tested previously - it's possible that the new idea is slightly different, or you may decide that it is the same and try testing another idea, or you could decide that your product is substantially different, and the same idea may be worth testing again. To make this decision, it is essential to capture not just the experiment results but the broader context of what your product looked like at the time and the test variants. Getting your team to document is always a challenge. GrowthBook takes two approaches to help with this. The first is to make it super easy to add documentation directly in the platform you’re already using for the experiment. Secondly, we added launch checklists, which require completing certain tasks before your team is able to start an experiment. ### Custom Fields Enterprise users can add custom fields to features and experiments. For example, these custom fields can be used to require links to Jira tickets, or to capture additional information that is important to your process or organization. Custom fields can be added from the **Settings** → **Custom Fields** page. Custom fields can be set to required or optional, and scoped to specific projects. Currently, custom fields has support for URL, text, textarea (larger text), markdown, number, enum, multiselect, date, and boolean types. ![Custom Fields Page](/images/using/custom-fields-page.png) ### Custom Markdown Enterprise users can inject custom markdown content into key pages of the GrowthBook app. This feature allows you to provide organization-specific guidance on processes for setting up experiments and features. Additionally, we've made certain Handlebars variables available on each page. This enables you to use [Handlebars templates](https://handlebarsjs.com/guide/#simple-expressions) to dynamically customize your messaging based on context. For example, writing ``` {{#if tags}} Guidance for experiments with tags... {{/if}} ``` in markdown for the Experiment page becomes `Guidance for experiments with tags...` when the experiment has at least one tag. | Page | Variables | | ---------------- | -------------------------------------------------------------------- | | Experiments List | `user` `orgName` | | Experiment | `user` `orgName` `experiment` `experimentStatus` `tags` | | Features List | `user` `orgName` | | Feature | `user` `orgName` `featureKey` `featureType` `tags` | | Metrics List | `user` `orgName` | | Metric | `user` `orgName` `metricName` `metricType` `metricDatasource` `tags` | To view and edit custom markdown, go to **Settings** → **General** → **Custom Markdown** → **View Custom Markdown Settings**. ## Searching GrowthBook has a powerful search feature that allows you to quickly find the feature, experiment, or metrics. By default, text searches with this search input will search based on the name, description, and other meta information. You can also search using syntax search to search for specific fields. ### Syntax Search Syntax search allows you to search for specific fields in GrowthBook. The syntax search allows for exact matching, starts with, greater, less, and contains. You can also negate any of the operators using !. Syntax searches are constructed in the format of `[field]:[operator][value]`. You can also combine multiple fields using the same syntax. For example, `name:~pricing status:running` will search for all running experiments with the name containing the string "pricing". | Syntax operator | Description | | --------------- | ------------------------- | | : | exact match | | := | exact match | | :~ | contains | | :^ | starts with | | :> | greater than | | :\ | not greater than | | :!\archivedIf the experiment is archiveddraftThe experiment is in draft statusrunningThe experiment is runningstoppedThe experiment is stoppedwonThe experiment been marked as wonlostThe experiment has been marked as lostinconclusiveThe experiment has been marked as inconclusivevisualThe experiment has variants made with the visual editorredirectThe experiment has a URL redirect experiment | | has | Supports searching on the experiment states. Supported fields projectThe experiment belongs to at least one projectvisualChangeThe experiment has a visual editor changeredirect/redirectsThe experiment has a URL redirectfeature/featuresThe experiment has features attachedhypothesisThe experiment has a hypothesisdescriptionThe experiment has a descriptionscreenshotsThe experiment has at least one screenshottempRolloutThe experiment has an active temporary rollout | #### Examples ``` name:~pricing status:running ``` Show all running experiments with the name containing the string "pricing" ``` tag:checkout result:won updated:>2024-06-05 ``` Show all experiments tagged with "checkout" that have been marked as won and were updated after June 5th, 2024 ``` is:archived has:visualChange variations:>2 ``` Show all archived experiments that have a visual change and have more than 2 variations in the experiment ``` owner:patty has:!hypothesis ``` Show all experiments owned by Patty that do not have a hypothesis ``` created:2) | | tag | Features tagged with this tag | | project | The feature's project | | version or revision | The feature's revision number | | experiment | The feature is linked to the specified experiment | | created | The feature's creation date, in UTC. Date entered is parsed so supports most formats. (eg: created:\>"2024-06-05" or created:\archivedIf the feature is archiveddraftThe feature is in draft statusstaleThe feature is stale | | has | supports searching on the feature states. Supported fields projectThe feature belongs to at least one projectdraft or draftsThe feature has a draftprerequisites or prereqsThe feature has at least one prerequisite flagsvalidation or schema or jsonSchemaThe feature has a JSON schema attachedrule or rulesThe feature has at least one ruleexperiment or experimentsThe feature has at least one experiment rulerollout or percentThe feature has at least one percentage rollout ruleforce or targetingThe feature has at least one force rule | #### Examples ``` key:~pricing on:production ``` Show all features with the key containing the string "pricing" that are `on` in the production environment ``` tag:checkout stale:true ``` Show all features tagged with "checkout" that are stale ``` is:archived has:prerequisites ``` Show all archived features that have prerequisites ``` owner:abby has:draft rules:0 ``` Show all features owned by Abby that have a draft and have no rules ``` updated:>"2024-06-05" has:experiment ``` Show all features that have experiment rules and were updated after June 5th, 2024 ### Metrics Syntax Fields | Syntax field | Description | | :----------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | The metric name (eg: name:~revenue) | | id | The metric's id (eg: id:met_abc123) | | type | The type of metric. Either `proportion`, `mean`, `ratio`, or `quantile`. We also support some legacy naming if you prefer to search with that: `binomial`, `count`, `revenue`, and `duration`. | | owner | The creator of the metric (eg: owner:pat) | | tag | Search by metric tag tag | | project | The metric's project | | datasource | Metrics that use a specific data source (eg: datasource:bigquery) | | created | The metric's creation date, in UTC. The date entered is parsed so supports most formats | | updated | The date the metric was last updated, in UTC. The date entered is parsed so supports most formats | | is | supports searching on the metric properties. Supported fields archivedIf the metric is archivedofficialIf the metric is Official and managed by the API or a config file instead of the GrowthBook UIfactIf the metric is defined on top of a Fact Table | | has | supports searching on various metric properties. Supported fields projectsThe metric is scoped to specific projectstagsThe metric has at least one tagdatasourceThe metric has a datasource | #### Examples ``` type:ratio name:~session ``` All ratio metrics where the name contains "session" ``` tag:checkout is:fact ``` All fact metrics with the tag "checkout" ``` updated:>"2024-06-05" owner:jeremy ``` Show all metrics owned by "jeremy" that were updated after June 5th, 2024 --- ## Using GrowthBook # Guide on using GrowthBook ## Introduction In today's data-driven world, businesses of all sizes rely on A/B testing to make data-driven decisions. A/B testing, also known as split testing, has come a long way from a simple tool to optimize websites, and is often used as a powerful tool to determine the impact of any changes to your application. By measuring how your user behavior and engagement changes in a controlled manner, you can determine causally if your hypothesis is correct, and make informed data-driven decisions that improve user experience, increase conversions, and drive growth. This document is intended to be an open source and continuously updated guide to A/B testing. Whether you're a seasoned expert at running experiments, or just starting out, this book will provide you with the knowledge and skills you need to run a successful A/B testing program, with a specific focus on GrowthBook, an open source feature flagging and A/B testing platform. In the following chapters, we'll start with an overview of what A/B testing is, and familiarize you with the terms that are commonly used. We'll cover the basics of statistical significance, sample size, and other key concepts that are essential for understanding A/B testing. Next, we'll cover the best practices for running an A/B test, followed by some of the common mistakes and pitfalls that can affect experiment programs. Finally, we'll go beyond individual A/B tests and talk about how to run an experimentation program, and then specifics of how to do this well with GrowthBook. We hope after reading this guide, you'll understand that A/B testing is a critical tool for determining causal impact of the changes you make, as well as optimizing flows. By making informed data-driven decisions, you can improve user experience, increase conversions, and drive growth. With the open source A/B testing tool, GrowthBook, you have a powerful and flexible platform that can help you run experiments quickly and easily. We hope that this guide will give you the knowledge and skills you need to run a successful A/B testing program and make data-driven decisions. Whether you're a developer, product manager, data scientist, marketer, or business owner, A/B testing can help you achieve your goals and drive growth. ## Contents - [Fundamentals of AB Testing](/using/fundamentals) - [Experimentation Best Practices](/using/experimentation-best-practices) - [Experimentation Common Problems](/using/experimentation-problems) - [Experimentation As Part of Your Development Process](/using/product-development) - [Experimenting in GrowthBook](/using/experimenting) - [GrowthBook Organization Best Practices](/using/growthbook-best-practices) - [Securing GrowthBook](/using/security) - [Experimentation Programs](/using/programs) ## Other resources At GrowthBook we highly recommended the book: "Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing" by Ron Kohavi, Diane Tang, and Ya Xu. It is available on Amazon or on Ronny's site at [https://www.exp-platform.com/Documents/GuideControlledExperiments.pdf](https://www.exp-platform.com/Documents/GuideControlledExperiments.pdf) --- ## Experimentation-driven product development # Experimentation-driven product development Experimentation-driven product development is a shift in product development from focusing **shipping** new products, to focus on shipping features that have an **impact** on your business. The best way of determining impact is through A/B testing. _The ideal goal with product-driven experimentation is to test every new feature that is developed._ This level of experimentation often requires adjusting your existing product process. If you want to read more about how to make the case for experimentation-driven product development, or the benefits to your culture, you can read our section on [experimentation programs](/using/programs). ## Platform integration Experimentation-driven product development requires a tight integration between your product and the experimentation platform. This is important to keep the incremental cost of running an experiment low, while increasing the ability to run a high volume of experiments. The cost can be cost in terms of effort, and cost in terms of actual money. You want to keep these a low as possible and make sure your platform encourages your team to run experiments. Given this, many companies choose to build their own experimentation platform. This is a big undertaking, and much harder than it might seem at first. There is also ongoing maintenance costs, as well as the risk in making product decisions on a platform that might have undiscovered bugs. ### Reducing costs Costs of running experiments can be broken down into a few categories: the cost of data storage, the cost of engineering time, and the cost of the platform itself. GrowthBook is designed to address these costs directly and allow you to run a lot of experiments. GrowthBook is warehouse native and uses any data you already have. It was also designed to be extremely easy to add an experiment (2 lines of code) and have a high-quality developer experience that reduces engineering costs. GrowthBook itself is open-core and extremely economical to run. ## Product prioritization changes As you become aware of HiPPOs effect on your decision-making process and start to move away from this, you need another way to prioritize projects. There are many prioritization frameworks to help with this (we have a [whole section on experimentation prioritization frameworks](/using/programs#prioritization)), but the goal of the system you choose should be to encourage building smaller testable features, a high frequency of experiments, and to make sure you have a good mix of ideas and sizes of projects. This is a big change from the traditional product development process, where we would spend a lot of time trying to predict what features will work, and then spend a lot of time building them. With experimentation-driven product development, we spend less time predicting and guessing and more time testing. This means that we can spend less time building features that don’t work, and more time building features that do work. ### HAMM HAMM is a framework to help you think about experimentation-focused product development. It stands for Hypothesis, Actions, Measure, and MVP. It is one product framework to help increase your learning rate, and build a culture of experimentation. - **Hypothesis**: What is the hypothesis you are testing? This should be a clear, falsifiable statement of what you are trying to learn. See our section on how to make good [hypothesis](/using/fundamentals#hypothesis-1). - **Actions**: What are the actions you expect your user to take if this hypothesis is true? - **Measure**: What are the metrics that you could measure that would indicate that the user is doing the actions you expect? What might be a counterfactual metric that would indicate that the user is not doing the actions you expect? What are the guardrail metrics that you want to make sure you don't negatively impact? - **MVP**: Given the above, what is the smallest thing you could build to test this hypothesis? Thinking about the HAMM process at the beginning of a project lays the groundwork for a high-quality experiment. You'll have the hypothesis, the metrics, and the success criteria (OEC). With a smaller MVP or MTP (Minimum Testable Products), you can also increase your experimentation rate and therefore your learning rate. ## Why you're not seeing experiment impacts Quite often, companies run A/B tests that show positive results, and yet when overall metrics are examined, the impact of these tests is invisible. You might be expecting to see inflection points around the time the experiment was implemented. Here are some reasons why: - **Confusing statistics** - Interpreting results can be confusing without a solid grasp of what the statistics are telling. - **Bad practices** - You may be running experiments that are not valid, or are not measuring the right thing. See the Peeking problem. - **Lost in the noise** - The impact of the experiment may be too small to be visible. This is especially true if you are running a lot of experiments - but this is not bad, just hard to see on a macro level. - **Optimizing wrong product** - You might be experimenting with a section of your product that doesn't represent a large fraction of your overall use. Even if you're successful in these areas of the product, the overall impact will be limited by the small fraction of users that you've affected. - **Optimizing wrong metric** - You might be optimizing for a metric that doesn't matter. For example, you might be optimizing for a metric that is not correlated with revenue, your main KPI. This is especially true if you are optimizing for a proxy metric, such as clicks, instead of the actual metric, such as revenue. You can read more about this topic on our blog post [Why the impact of A/B testing can seem invisible](https://medium.com/growth-book/why-the-impact-of-a-b-testing-can-seem-invisible-5b2d69efa48) --- ## Experimentation Programs # Experimentation Programs “Experimentation”, or being more “data driven” can mean a lot of different things for different companies. It can be anything from running 1 test a quarter, to running 10s of thousands of experiments simultaneously. This difference of experimentation sophistication can be thought of with the crawl, walk, run, fly framework (From “Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing”, Ron Kohavi, Diane Tang, Ya Xu). **CRAWL - Basic Analytics** Companies at this stage have added some basic event tracking and are starting to get some visibility into their users behavior. They are not running experiments, but the data is used to come up with insights and potential project ideas. **WALK - Optimizations** After implementing comprehensive event tracking, focus now turns to starting to optimize the user experience on some parts of the product. At this stage, A/B tests may be run manually, limiting the number of possible experiments that can be run. Typically at this stage, depending on the amount of traffic you have, you may be running 1 to 4 tests per month. **RUN - Common Experimentation** As a company realizes that experimentation is really the only way to causally determine the impact of the work they are doing, they will start to ramp up their ability to run A/B tests. This means adopting or building an experimentation platform. With this, all larger changes made to their product are tested, as well they may have a Growth Team that is focused on optimizing parts of their product. At this stage, a company will be running 5 - 100 A/B Tests/Month. This may also include hiring a data team to help with setting up and interpreting the results. **FLY - Ubiquitous Experimentation** For companies that make it to the flying stage, A/B testing becomes the default for every feature. Product teams develop success metrics for all new features, and then they are run as an A/B test to determine if they were successful. At this point, A/B tests are able to be run by anyone in the product and engineering organization. Companies at this stage of ubiquitous experimentation can run anywhere from 100 to 10,000+ A/B Tests/Month. ## Making the case for experimentation If your organization doesn't yet experiment often, you may need to make the case for why you should. The best way, when you are working on a project, is to ask your team "what does success look like for this project?" and "How would we measure that success?" In this case, two things will happen, either they'll give an answer that is not statistically rigorous, like looking at the metrics before and after, or they will say some variation of "We don't know". Once you're team realizes that A/B testing is a controlled way to determine causal impact, they'll wonder how they ever built products without it. The next pushback you may get is that A/B testing is too hard, or that it will slow down development. This is where you can make the case for GrowthBook. GrowthBook is designed to make A/B testing easy, and to make it so that you can run experiments without slowing down development. We are warehouse native, so we use whatever data you already are tracking, and our SDKs are extremely light weight and developer friendly. The goal at GrowthBook is to make it so easy and cost efficient to run experiments you'll test far more often. You can watch a video of making the case for AB testing here: ### Why AB test? - **Quantify Impact** You can determine the impact of any product change you make. There is a big difference between "we launched feature X on time" and "we launched feature X on time and it raised revenue by Y". - **De-risking** You can de-risk any product change you make with A/B testing. You can test any change you make to your product, and if it doesn't work, you can roll it back. Typically new projects, if they are going to fail, will fail in 3 ways: The project has errors, the project has bugs that unexpectedly effect your metrics/business, or the project has no bugs or errors, but still negatively effects your business. A/B testing will catch all of these issues, and allows you to roll out to a small set of users to limit the impact of a bad feature. - **Limiting investment on bad ideas** As we discussed in our HAMM section - When you focus on building the smallest testable MVP (or MTP) of a product, you can save a lot of time and effort put into a bad idea. You build the MVP and get real users testing it, and if it turns out that you cannot validate the hypothesis behind the idea, then you can move on to other projects, and limit the time spent on ideas that don't work or that will have a negative impact on your business. - **Learning** If you have a well-designed experiment, you can determine causality. If you limit the number of variables that your test has, you can know the exact change drove that change in behavior, and apply these learnings to future projects. ### Why A/B testing programs fail - **Lack of buy-in** If you don't have buy-in from the top, it can be hard to get the resources you need to run a successful experimentation program. You'll need to make the case for why you should experiment, and why you need the resources to do so. - **High cost** Many experimentation systems, especially legacy ones, can be expensive to run or maintain. When the costs are high, you can end up running fewer experiments, and with fewer experiments, the impact is lower. Eventually, a program in this state can atrophy and die. - **Cognitive Dissonance** As you're often getting counter-intuitive results with A/B testing, team members can start to question the platform itself, and may prefer to listen to their gut over the data. This is why building trust in your platform is so important. - **No visibility into the program's impact** Without some measure of the impact of your experimentation program, it can be hard to justify the expense of running it. You'll want to make sure you have a way to measure the impact of your experimentation program. ## Measuring Experiment Program Success Once you have added an experimentation program, teams often look for a way to measure the success of that program. There are a few ways you can use to measure the success of your experimentation program, such as universal holdouts, win rate, experimentation frequency and learning rate. Each of these has their own advantages and disadvantages. ### Universal Holdouts Universal holdouts is a method for keeping a certain percentage of your total traffic from seeing any new features or experiments. Users in a universal holdout will continue to get the control version of every test for an extended period of time, even after an experiment is declared, and then those users are compared to users who are getting all the new features and changes. This effectively gives you a cohort of users that are getting your product as it was, say 6 months ago, and comparing it to all the work you’ve done since. This is the gold standard for determining the cumulative impact of every change and experiment, however, it has a number of issues. To make universal holdouts work, you need to keep the code that delivers the old versions running and working on your app. This is often very hard to do. Some changes can have a non zero maintenance cost, block larger migrations, or limit other features until the holdout ends. Also, any bugs that arise that only affect one side of the holdouts (either control or in the variations), can bias the results. Finally, due to the typically smaller size of the universal holdout group, it can take longer for these holdout experiments to reach significant values, unless you have a lot of traffic. Given the complexity of running universal holdouts, many companies and teams look for other proxy metrics or KPIs to use for measuring experimentation program success. ### Win Rate It can be very tempting to want to measure the experimentation win rate, the number of A/B tests that win over the total number of tests, and optimize your program for the highest win rate possible. However, using this as the KPI for your experiment program will encourage users to not run high risk experiments and creates a perverse incentive for more potentially impactful results (see Goodhart’s Law). Win rate can also hide the benefits of not launching a “losing” test, which is also a “win”. ### Experimentation Frequency A more useful measure than win rate is optimizing for the number of experiments that are run. This encourages your team to run a lot of tests which increases the chances of any one test producing meaningful results. It may, however, encourage you to run smaller experiments over larger ones, which may not be optimal for producing the best outcomes. ### Learning Rate Some teams try to optimize for a “learning rate” which is the rate at which you learn something about your product or users through A/B testing. This does not have the frequency or win rate biases, but also is nebulously defined. How do you define learning? Are there different qualities of what you learn? ### KPI Effect If you can pick a few KPIs for your experimentation program, you should be able see the effects of the experiments you run against this. You may not be able to see causality precisely due to the natural variability in the data, and typically small improvements from an A/B test, but by aligning by the graph of this metric to experiments that are run, you may start to see cumulative effects. This is what GrowthBook shows with our North Star metric feature. ## Prioritization Given the typical success rates of experiments, all prioritization frameworks should be taken with a grain of salt. Our preference at GrowthBook is to add as little process as possible and to maximize for a good mix of iterative and innovative ideas. ### Iteration vs Innovation It is useful to think of experiment ideas on a graph with one axis being the effort required and the potential impact on the other. If you divide the ideas into two for high effort/impact and low effort/ impact, you’ll end up with the following quadrant. Low impact High impact High effort Danger Prioritize Low effort Prioritize Run now The low effort, high impact ideas you should be running immediately, and similarly the high effort, low impact ideas you may not want to run at all. But this leaves the other two, low effort but low impact (smaller tests), and high effort high impact ideas (big bets). If you over index for smaller test ideas, you can increase your experimentation frequency, but risk not getting larger gains. If you over index for bigger bets, you decrease your experimentation frequency at the hope of larger returns, at the risk of not achieving the smaller wins which can stack up. You can also consider the smaller tests as being “iterative” and the bigger bets as “innovative”. Finding a good mix of small, iterative tests and bigger bets/innovative tests is the best strategy. What constitutes “good” is up to the team. Some companies will bucket their ideas into these two groups, and then ensure that they are pulling some percentage of ideas from both lists. A healthy mix of large and small ideas are important to a successful experimentation program. ### Prioritization frameworks In the world of A/B testing, figuring out what to test can be particularly challenging. Often prioritization requires a degree of gut instinct which is often incorrect (see success rates). To solve this, some recommend prioritization frameworks, such as ICE and PIE. :::note ICE and PIE frameworks Note: Please keep in mind that while these frameworks may be helpful, they can work to give the appearance of objectivity to subjective opinions. ::: #### ICE The ICE prioritization framework is a simple and popular method for prioritizing A/B testing ideas based on their potential impact, confidence, and ease of implementation. Each idea is evaluated on each of these factors and scored on a scale of 1 to 10 and then averaged to determine the overall score for that idea. Here's a brief explanation of the factors: - **Impact**: This measures the potential impact of the testing idea on the key metrics or goals of the business. The impact score should reflect the expected magnitude of the effect, as well as the relevance of the metric to the business objectives. - **Confidence**: This measures the level of confidence that the testing idea will have the expected impact. The confidence score should reflect the quality and quantity of the available evidence, as well as any potential risks or uncertainties. - **Ease**: This measures the ease or difficulty of implementing the testing idea. The ease score should reflect the expected effort, time, and resources required to implement the idea. To calculate the ICE score for each testing idea, simply add up the scores for Impact, Confidence, and Ease, and divide by 3: > ICE score = (Impact + Confidence + Ease) / 3 Once all testing ideas have been scored using the ICE framework, they can be ranked in descending order based on their ICE score. The highest-ranked ideas are typically considered the most promising and prioritized for implementation. #### PIE Like the ICE Framework, the PIE framework is a method for prioritizing A/B testing ideas based on their potential impact, importance to the business, and ease of implementation. Each score is ranked on a 10 point scale. - **Potential**: This measures the potential impact of the testing idea on the key metrics or goals of the business. The potential score should reflect the expected magnitude of the effect, as well as the relevance of the metric to the business objectives. - **Importance**: This measures the importance of the testing idea to the business. The importance score should reflect the degree to which the testing idea aligns with the business goals and objectives, and how critical the metric is to achieving those goals. - **Ease**: This measures the ease or difficulty of implementing the testing idea. The ease score should reflect the expected effort, time, and resources required to implement the idea. To calculate the PIE score for each testing idea, simply multiply the scores for Potential, Importance, and Ease together: > PIE score = Potential x Importance x Ease Once all testing ideas have been scored using the PIE framework, they can be ranked in descending order based on their PIE score. The highest-ranked ideas are typically considered the most promising and prioritized for implementation. ### Bias in prioritization Regardless of what prioritization method you choose, it's quite common to develop a bias for a particular types of ideas within a team. Make sure you're open to ideas that may not fit your preconceived notions of what will work (see [Semmelweis Effect](/using/experimentation-problems#semmelweis-effect)). Be mindful of when you're saying "no" to an idea if it's based on data or opinion. The goal, in the end, is to improve your business by producing the best product. ## Experimentation Culture Adopting experimentation as a key part of being a more data-driven organization has numerous benefits to culture. Specifically around areas of alignment, speed, humility, and collaboration. ### Alignment Adopting a north star metric or KPI that would drive our business success removes a lot of ambiguity about projects because we had clear success metrics. By making sure you have defined success metrics at the start of your planning cycle, you achieve alignment around your goals. This helps reduce the invariable scope creep and pet features from inserting themselves — or at least gives you a framework to say “yes, but not now.” Knowing what success means also allows developers to start integrating the tracking needed to know if the project would be successful from the beginning, which can often be forgotten or only done as an afterthought. ### Speed When adopting an experimentation mindset, the default answer to a difference of opinion becomes “let’s test it” instead of long drawn out ego bruising meetings. This helps reduce personal opinions or bias affecting decisions. Quite often decisions in companies without this mindset are made by whomever is the loudest, or the HiPPOs (Highest Paid Person’s Opinion). By focusing on which metrics defined success, and defaulting to running an experiment, you can remove the ego from the decision process, and move quickly. Experimentation can also help increase your product velocity by minimizing the time it takes to determine if your new product or feature has product market fit. Most big ideas can be broken down into a small set of assumptions that, if true, would mean your larger idea may be successful. If you can prove or disprove these ideas, you can move more quickly and not waste time on failing ideas (loss avoidance). ### Intellectual humility AB testing shows us that, in most of the cases, people are bad at predicting user behaviors. When you realize that your opinions may not be correct, you can start to channel your inner Semmelweis and be open to new ideas that challenge any deeply held entrenched norms or beliefs. Having an open mind and intellectual humility for new ideas can make your workplace a more collaborative environment, and produce better products. ### Team collaboration When you are open to new ideas, you can remove the silos that prevent teams from collaborating well. The goal is to produce the best product as measured by a specific set of metrics. With this alignment, and the openness to new ideas, you can dramatically increase collaboration as good ideas come from anywhere. ## Driving Experimentation Culture Developing a culture of experimentation can be hard, especially in a company where it has never existed. It requires a lot of buy-in from the top down, and/or a lot of evangelism from the bottom up. ### Top down This is often the easiest way to drive experimentation culture. If the CEO or CTO or CPO says that they want more experimentation, they can make it happen. In these situations, picking the right platform and educating your team becomes the hardest part. You'll want to pick a platform that the developers like to use, that doesn't add unnecessary effort per experiment, and that brings the incremental cost per experiment close to zero. These are some of the reasons we built GrowthBook. If you do decide on GrowthBook, we can also help with educating your team. ### Bottom up If you don't have buy-in from the top, you can still drive experimentation culture from the bottom up. Typically this starts with one team that wants to start experimenting. They may start with a simple test. Experimentation like this can be contagious, and other teams may start to see the benefits of running experiments. It's important with this approach to make sure that you are sharing your results, both good and bad, and that you are evangelizing the benefits of experimentation. ### Sharing One great way to get fresh ideas and to help experimentation culture is to share your experiment ideas and results. Our preferred way to present your results is with an experiment review meeting. The premise behind these is to talk about the experiment without revealing the results, and to have people guess about the outcome. Specifically, you talk about the hypothesis and observations as to what and why you are testing, and then talk about the metrics you’re testing against, and then show screen shots of the variations (if applicable). You can have people vote simply by raising their hand. Once you’ve had people guess, you reveal the actual results. This is a great way to help build intellectual humility and also collect new ideas. GrowthBook has built experiment review meetings directly into the platform. You can create presentation from the management left navigation. You can then share the presentation with your team, and they can vote on the results. ## Organizational Structures As you start to scale your experimentation program, you’ll want to think about how you want to organize your teams to ensure high frequency and high quality. There are a number of different ways to organize your teams, and we’ll go through some of the most common ones we’ve seen. ### Isolated teams, When companies first start experimenting with experimenting, they often start with isolated teams. This can even be one individual on a team. One of the problems with this approach is that as an individual, it is hard to have good ideas to test continually, and you may suffer from idea bias, where your experiences and expertise limit the number and type of ideas you test. Another issue is that successes and failures are not shared. As is typical of experiment programs, if you present ideas that are failing at a 60%+ rate, people may think that the team is doing something wrong. These isolated teams can be critical in helping grow awareness of experimentation-driven development. However, the isolated team does not scale well, and running the frequency of experiments to see large impacts will be hard. If the team and leadership like the results, you’ll want to expand to one of the other structures. ### Decentralized Teams As awareness of the ease of and insights gained through experimentation, more teams may start experimenting. This is great and increases the frequency of experimentation that you can run. Each team is empowered to design and start their own experiments- this is sometimes referred to as experimentation democratization. There can be some downsides with this approach. It can end up like the Wild West, where best practices, data, metrics, and tooling may not be shared from team to team. This can make it hard for teams to ensure consistent quality and trustworthiness of the results. ### Center of Excellence To compensate for the problems of decentralized experimentation programs, many companies will switch to a center-of-excellence approach. With this structure, a central experimentation team ensures that experiments follow best practices, have a testable hypothesis, and have selected the right metrics before launching. This team can also ensure that the data looks right as it comes in and that the results are interpreted correctly. One of the issues with the center-of-excellence approach is that it can easily become a bottleneck of excellence and limit the number of experiments that are run. ### Hybrid Combining the best of the decentralized teams and center-of-excellence is one of the best ways we’ve seen to run experimentation programs. The Hybrid approach involves an experimentation team that oversees the experiments that are run but don’t directly gatekeep the launching of experiments. In this role, the experimentation team serves as advisors to the teams that are running experiments, helps them improve the quality of experiments, and can also help look into any issues that appear. They can also ensure that the platform, metrics, and data are following their standards. This approach aims to have the experimentation team help educate the product teams on best practices and common pitfalls with running experiments. --- ## Security # Security GrowthBook is built with security in mind, and we have made architectural decisions to ensure that your GrowthBook instance can be as secure as possible. You can read about some of our internal security practices [here](https://www.growthbook.io/security) and on [GitHub](https://github.com/growthbook/growthbook/blob/main/SECURITY.md). This document covers some of the security considerations for administrators when setting up and using GrowthBook. ## Data Security GrowthBook only stores account information for your users with GrowthBook accounts (email and name). For feature flagging, evaluations typically happen within the SDK, so none of your user’s information is sent to GrowthBook. For experiment reporting, we connect to your data warehouse to pull the assignment/exposure and metric information. This data remains in your data warehouse; GrowthBook only stores the aggregate results, such as the total number of users exposed to each variation and other statistical information. No PII is stored or transferred to GrowthBook with the experimentation reports. There are some ways where personal information may be stored or exposed with GrowthBook. If you are using GrowthBook on the client side of your application, the rules about how each feature will be exposed to your users are publicly accessible by inspecting network requests. Usually, these rules contain no personal information, but if you are targeting a specific user or set of users, then this information may be visible to malicious users. If you use GrowthBook on the server side, this information is not exposed to the client. For these reasons, targeting based on PII when using GrowthBook client side is not recommended. If you have to target based on PII on the client side, GrowthBook has some ways to make this secure. You can use hashed attributes, where the values are hashed before sending, or you can use encrypted attributes, where the payload is encrypted before sending to the client. Keep in mind that encrypted SDK endpoints will have to decrypt the payload before use, and that means if you are using GrowthBook client side, a malicious actor can see the decrypted payload. You can enable encryption when setting up the SDK, and you can select an attribute as 'hashed' when creating the attribute. Finally, to avoid these issues, you can also use 'remote evaluations' to evaluate flags based on sensitive information without exposing it, even on the client side. With remote evaluations, the SDK will send the user attribute to your server (or ours), and then that attribute is matched against the rules, and then the state of the feature is returned without revealing the targeting rules to the client. The downside of this approach is that each evaluation requires a network request to evaluate the feature, which can slow down your application. ## Data Access GrowthBook requires read-only access to your data warehouse. This connection information is kept encrypted. You can use permissions to help protect access to your data warehouse through the GrowthBook UI. You can assign users who can edit the connection info, SQL queries for the assignment, or metric queries. Data sources can also be scoped to projects to help isolate access to your data warehouse. You can adjust the permissions so that only users who have access to the project and have the proper permission levels can edit those queries. If you require more separation of your data and metrics within GrowthBook, you can create separate cloud organizations, or run GrowthBook in multi-tenant mode (available when self-hosting as part of GrowthBook Enterprise). With this, you can have separate GrowthBook organizations running from one GrowthBook instance. Each organization will have its own data source, metrics, and users. There is a super-admin account type that can manage users across organizations. ## Infrastructure Security GrowthBook Cloud is hosted on AWS, in a multi-tenant environment. We use industry-standard security to ensure that your data is secure. If you require additional security, we also offer self-hosted options. When you self-host, no data leaves your infrastructure. We do have anonymous usage tracking enabled by default, but this can be disabled (see [self-hosted](/self-host)). ## Self-hosted deployments GrowthBook can be self-hosted on your own infrastructure. If self-hosting, we recommend that you keep GrowthBook behind a firewall, and accessible via a VPN. See [self-hosting](/self-host) for more information. GrowthBook should also be regularly updated to ensure that you are running the latest version with the latest security patches. GrowthBook updates are backward compatible and can be easily applied with a single command. See [updating GrowthBook](/self-host#updating-to-latest) Before deploying GrowthBook in production, we recommend that you make sure you've configured GrowthBook correctly: - Change the `JWT_SECRET` environment variable. This is used to sign the JWT tokens used for authentication, and needs to be changed from the default. - Change the `ENCRYPTION_KEY` environment variable. This is used to encrypt sensitive data, and should be set to a long random string. - Set the `NODE_ENV` environment variable to `production`. This will enable add additional optimizations and disable some debugging features. ## Audit logs GrowthBook keeps an audit log of all actions taken on the platform. The audit logs is useful if you need to determine what actions as user has done. You can access this page from the Settings → Log in the left navigation. You can read more about audit logs [here](/account/audit-logs). --- ## AI Mode — Generate A/B Test Variations With AI import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; AI mode is the fastest way to draft an A/B test variation in the [Visual Editor](/app/visual). You describe what you want in plain language ("make the hero headline more urgent and the primary button green"), and the AI proposes specific DOM-level changes — element selectors, new content, new styles — that you can preview, accept, edit, or reject one at a time. ![The GrowthBook Visual Editor AI Editor](/images/visual-editor/ve-ai-chat.png) It's especially good for: - Multi-property changes ("make this card more premium") without touching each control - Change text content without needing to find the right content field in the property panel - Less technical users who don't want to think about CSS selectors You can mix AI mode and [Manual mode](/app/visual/manual-mode) freely within the same variation — accept the AI's draft, then jump into Manual mode to fine-tune. ## Prerequisites Before AI mode is usable on your account: 1. **An org admin needs to configure an AI provider.** Go to **Settings → AI** in GrowthBook and add an OpenAI, Google, or xAI API key. See [AI Integrations](/integrations/ai) for the full setup. 2. **Your plan must include AI features.** AI mode is part of the `ai-suggestions` commercial feature. Pro and Enterprise plans include it; Free does not. 3. **You need to be connected** to your GrowthBook account in the [Visual Editor side panel](/app/visual/install-and-connect). If AI mode is greyed out in the side panel, the most common cause is that no AI provider key is configured yet for the organization. ## Opening AI mode After you open the side panel and target a URL, you'll see two top-level tabs: **AI** and **Manual**. AI is selected by default. The AI panel has three parts: - A **prompt box** at the top of the panel where you describe what you want. - A **chat log** that fills in as you make requests — each turn shows what you asked, followed by an assistant card with the proposed changes. - The **Changes panel** at the bottom of the side panel, which lists every change accepted so far for the current variation. When the chat log is empty, the prompt area also surfaces a few **suggested prompts** and **quick actions** to help you get started. ## Writing good prompts The AI does its best work when prompts are concrete about _what_ and _where_, but loose about _how_. **Good:** - `Change the hero headline to something more urgent and benefit-led` - `Make the primary CTA button bright green and a little larger` - `Hide the secondary nav links on the right` - `Swap the hero image for one showing a person using a laptop in a coffee shop` - `Give me three alternative headlines for the product card` **Less good:** - `Make the page better` — too vague; the AI will guess and the result will be unfocused. A few tips that consistently improve results: - **Name the element by what it is, not how it's coded.** "The hero headline" works; ".hero h1" rarely needs to be in the prompt. - **Describe intent.** "More urgent," "more premium," "more playful" gives the AI a direction. - **Ask for variants.** "Give me three options" or "draft a few alternatives" produces multiple suggestions in one turn. - **Chain prompts.** A second prompt can refine the first ("the green is too neon — make it more muted"). When naming an element in text isn't enough, use **Select element** to pin it — see [Adding element context](#adding-element-context-to-a-prompt). :::tip "Set the title" targets the H1 If you say something like "set the title to X," the AI targets the visible `` (or the most prominent heading) — not `document.title`. To change the browser tab title, say so explicitly: "change the page's browser tab title to X." ::: ## Adding element context to a prompt Click **Select element** (`@`) below the prompt box, then click an element on the page. It appears as a chip above the prompt — you describe _what_ to change; the chip tells the AI _where_. Pin multiple elements for batch edits; remove a chip with **×**. Adding element context to a prompt. Use this when the page has repeated or ambiguous targets (two headlines, identical cards, several "Submit" buttons). **Example:** pin the hero headline, then prompt _"Make it more urgent and benefit-led"_ instead of _"make the headline more urgent"_. For follow-up prompts on the same element without re-pinning, see [Multi-turn conversations](#multi-turn-conversations). ## Reviewing proposed changes When the AI returns, it adds an assistant card to the chat log. The card contains: - A short summary of what was changed and why. - One or more **diff cards** — each represents a single DOM change (a text edit, a style change, an attribute change, etc.). The diff card shows the element it targets and what's being set. Each proposed change is **previewed live on the page** as soon as it lands — you don't have to accept first to see the result. For each change you can: - **Accept** — the change moves into the Changes panel as part of the current variation. - **Reject** — the preview is reverted and the change is dropped. - **Edit** — open the change in Manual mode to tweak values before accepting. Reject doesn't end the conversation. You can ask for a different version ("more subtle — keep the original color, just bump the size") and the AI will produce a new proposal. ## Selector grounding When the AI proposes a change, it picks a CSS selector and validates it against the actual page DOM. If the selector doesn't match a real element, the change is rejected before it ever reaches you. This matters for two reasons: - **Reliability** — the AI can't propose changes against elements that don't exist or that match nothing. - **Stability** — the AI prefers stable selectors (semantic tags, stable classes, `data-*` attributes) over brittle ones (positional `nth-child` chains, random hash-suffix classes). The variation is more likely to keep working as your site evolves. If a proposal targets the wrong element, say so in the next prompt or pin the correct element and re-ask. ## Multi-turn conversations The chat log keeps the full thread for the current variation. You can layer changes: ``` You: change the hero headline to something more urgent AI: [proposes new headline] You: also make the headline larger AI: [proposes a font-size change on the same element] You: actually undo the size change and just bold it instead AI: [reverts size, proposes font-weight] ``` Each new turn sees the prior turns, so refinements work. If you want a clean slate, start a new variation. ## Image generation If you ask the AI to swap a hero image, generate product imagery, or anything else that requires a new image, AI mode hands off to the image-generation flow. You'll pick from one or more configured image models, optionally use the existing image as a visual reference, and accept the result into the changeset the same way as any other change. See [Images](/app/visual/images) for a full walk-through, including the list of supported image models and which ones accept reference images. ## What gets sent to the AI For each prompt, the back-end sends to the configured AI provider: - Your prompt text - A simplified snapshot of the relevant DOM (typically the elements visible in the viewport, plus any selected element) - The URL of the page being edited - Optionally, a small system prompt that biases the model toward stable, accessible changes Your GrowthBook experiment metadata, metric values, user data, and SDK payloads are **not** sent. The connection uses your org's configured AI provider key — see [AI Integrations](/integrations/ai) for the full data-sent inventory. ## Limits and caveats - **Single-page scope.** AI mode reasons about the page currently loaded in the Visual Editor's target URL. It can't propose changes against pages it hasn't seen. - **Selectors can still drift.** Even with selector grounding, large site rewrites or design-system changes can break previously-stable selectors. Run a sanity check on long-running visual experiments after major releases. - **Quality varies by model.** The org admin picks the underlying model in AI Settings — bigger/newer models generally produce better proposals at higher cost. The same prompt will produce different output across `gpt-4o-mini`, `gemini-2.5-pro`, and `grok-2`. - **AI mode is not a substitute for review.** Always preview the variation on a real page (and ideally a real preview link) before publishing. ## Next steps - **[Manual mode](/app/visual/manual-mode)** — fine-tune accepted changes. - **[Preview and QA](/app/visual/preview-and-qa)** — preview before publishing. --- ## Import from Figma or an Image — Design to Variant import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; :::info Review generated variations Generated results can vary, so always review the variation before you save it. Design → Variant also needs an up-to-date GrowthBook server and a configured vision-capable AI model. If your instance hasn't enabled it yet, the panel will say so and you can fall back to a mockup image (or ask an admin to update the server). ::: Design → Variant turns a **Figma frame** or a **mockup image** into a self-contained, testable variation. Instead of nudging the existing page element by element, you hand the AI a finished design and it rebuilds it as a **scoped component** (HTML + CSS, occasionally a little JS) and places it on the page as a variation. This is the fastest way to test a design your team already mocked up — a new hero, a redesigned pricing card, a promo banner — without hand-coding it. ## When to use it - ✅ A **self-contained section or component**: a hero, a feature card, a callout, a banner, a redesigned form. - ✅ You already have the design in **Figma** or as an **image** (a screenshot, an exported mockup, a design from any tool). - ⚠️ **Not** for full-page redesigns or whole-navigation overhauls. The AI is tuned to produce one in-page component; if the design is essentially a new page, it will tell you it's too large and decline rather than produce something broken. Break it into sections instead. ## Open the importer 1. Install and connect the [Visual Editor](/app/visual/install-and-connect) and open the side panel on the page you want to test. 2. In **AI mode**, click **Import from Figma or a mockup**. The **Design → Variant** panel opens. ## 1. Choose a source The panel has two source tabs: ### Figma link Paste a link to a **specific frame** — the URL must include a `node-id` (you get one by right-clicking a frame in Figma and choosing **Copy link to selection**). A link to a whole file without a frame selected won't work. Importing from Figma requires connecting your Figma account once (see [Connecting Figma](#connecting-figma) below). The connection is **read-only** and scoped to the file you link — GrowthBook never edits your Figma files. ### Mockup image No Figma needed. **Upload** a PNG, JPEG, or WebP (up to 8 MB), or **paste an image URL**. Anything works: a Figma export, a screenshot of a competitor, a Sketch/Photoshop mockup, even a whiteboard photo. ## 2. Place it on the page - **Place it in** — click **Pick element** and choose a container on the page (or type a CSS selector). This defaults to the whole page if you don't pick one. - **Placement** — how the generated component is positioned relative to the target: | Mode | Result | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | **Append inside** | Added as the last child of the target (the default). | | **Insert before** | Added immediately before the target, as its previous sibling. | | **Insert after** | Added immediately after the target, as its next sibling. | | **Replace contents** | Replaces everything inside the target. (Can't target ``/`` — that would wipe the page; pick a specific container.) | ## 3. Add instructions (optional) and generate Use **Extra instructions** to steer the result — e.g. _"use our brand blue for the button"_ or _"keep the existing headline copy."_ Then click **Generate variation**. The AI builds the component and previews it live on the page. Review it, then **accept** to save it into the variation, or refine your instructions and regenerate. Because the markup and styles are **scoped** to the generated component, they don't leak into or restyle the rest of the host page. :::tip Review before you save The AI reproduces the design as faithfully as it can, but fonts, spacing, and imagery can differ from the source. Treat the output as a strong first draft, eyeball it on the real page, and tweak with [Manual mode](/app/visual/manual-mode) or [global CSS](/app/visual/manual-mode) if needed before saving. ::: ## Connecting Figma The first time you use the Figma source, click **Connect Figma**. This opens Figma's authorization screen in a popup; approve it and you're returned to the side panel. - The integration requests a **single, read-only scope** (`file_content:read`) — enough to render the frame you link and read its layout/colors. It can't modify your files and can't see files you don't link. - The connection is **per-user** — each teammate connects their own Figma account, and tokens are stored encrypted. - You can **Disconnect** at any time from the panel. If you see _"Figma isn't set up on this GrowthBook instance,"_ the Figma integration hasn't been configured on your server yet. Use a **mockup image** instead, or ask an admin to set up the Figma OAuth app. (Self-hosted instances need the Figma OAuth credentials configured; GrowthBook Cloud has this managed.) ## Images and placeholders The AI reproduces **layout, typography, spacing, and color** from the design. For photographic imagery it can't recreate, it uses a neutral placeholder (a solid color or gradient from the palette) rather than hot-linking external assets. Swap those for real images afterward with the [image tools](/app/visual/images) — upload, AI-generate, or pick from your library. ## Limitations - **One component per generation.** Design → Variant produces a single scoped component, not a multi-section page. - **Too-large designs are declined.** If the frame is a full page or a large multi-section layout, the AI returns a message explaining it's too large instead of producing a partial result. Target a smaller container. - **Results vary.** Always preview and adjust before saving. - **Needs a vision-capable AI model.** Your org must have a vision-capable model configured in [AI Settings](/integrations/ai) (e.g. Gemini, GPT-4o/5, or Claude). Admins can set the model used for the Visual Editor there. ## Next steps - **[AI mode](/app/visual/ai-mode)** — describe changes in natural language for element-level edits. - **[Manual mode](/app/visual/manual-mode)** — fine-tune the generated component with the structured editor. - **[Images](/app/visual/images)** — replace the placeholders with real or AI-generated imagery. - **[Preview and QA](/app/visual/preview-and-qa)** — check the variation before publishing. --- ## Test Images in A/B Tests — Upload, AI Generation, and Library import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; Images are some of the highest-impact A/B tests you can run — hero photography, product imagery, illustration style. The [Visual Editor](/app/visual) gives you three ways to swap an image in a variation: 1. **Upload** — drop in a file from your machine. 2. **AI generation** — describe the image and let a configured AI model produce it, optionally using the existing image as a reference. 3. **Library** — pick from images your organization has already uploaded or generated, reusable across experiments and projects. All three paths land in the same `` (or ``) replacement and are managed identically in the [Changes panel](/app/visual/manual-mode#change-list). ## Replacing an image Open the [Visual Editor](/app/visual/install-and-connect) and target the page you want to edit. Select the `` element you want to replace. In Manual mode, the property panel shows an **Image** group with three tabs: **Upload**, **AI**, and **Library**. ### Upload tab 1. Click the upload area (or drag-and-drop a file onto it). 2. The new image appears as a live preview on the page, replacing the original. 3. Click **Use this image** to accept, or **Cancel** to discard. Supported formats are PNG, JPEG, WebP, and GIF. There's no hard size limit, but very large images slow page load — keep production assets under ~500 KB where possible. #### `` and `srcset` handling Modern image markup often wraps an `` in a `` element with multiple `` candidates and a `srcset` attribute for responsive resolution selection. If you just replace the `src`, the browser will keep picking the original from `srcset` and your variation won't show up. The Visual Editor handles this automatically: - The `srcset` attribute on the target `` is removed. - Any `` siblings inside a parent `` are removed. - The new image is the only candidate the browser sees. Reverting the image change (from the Changes panel) restores all of this — `srcset`, `` siblings, original `src` — to the pre-variation state. ### AI tab Type a description of the image you want and click **Generate**. The configured AI model returns one or more candidates that preview live on the page. Common useful prompt patterns: - `Photograph of a person using a laptop in a sunlit coffee shop, candid composition` - `Flat illustration of a chart trending upward, blue and orange palette` - `Product hero of a stainless steel water bottle on a white background, soft shadows` A few options: - **Use current image as reference** — appears when the selected model supports it. The AI uses the existing image as a visual cue for composition/style. Useful for "keep the same scene but with a different background color" type requests. - **Model picker** — appears when your org has more than one image model configured. See [available models](#available-image-models) below. - **Aspect ratio and size** — pulled from the original image so the replacement fits the existing layout. (You can still override by editing CSS dimensions in the Background & Border group.) When you're happy with a candidate, click **Use this image** to accept it into the variation. Generated images are stored in the [image library](#the-library-tab) so you can reuse them later. ### Library tab The Library tab shows every image your organization has uploaded or accepted from AI generation, across all experiments and projects. - **Grid view** of thumbnails with newest first. - **Click a thumbnail** to preview it as the new image; click **Use this image** to accept. - **Refresh icon** at the top to re-fetch the library — useful if a teammate just uploaded an image. - **Image count** in the header. Library images thumbnail-load lazily, so the panel stays responsive even with hundreds of images. #### Quarantine indicator AI-generated images that haven't been accepted into a variation yet show a small **orange dot** badge. These are temporary — they live in a quarantine bucket and the storage lifecycle removes them after 7 days. A short legend at the bottom of the library explains the dot. Why quarantine? Image generation produces a lot of throwaway candidates. We don't want every "let me try one more prompt" attempt to clutter your library forever. Once you accept a candidate into a variation, it gets promoted to permanent storage and the orange dot goes away. ## Available image models The model used for AI generation is set by your org admin in [AI Settings](/integrations/ai). The current options are: | Provider | Model | Reference image | | -------- | ------------------------------------ | --------------- | | Google | Gemini 2.5 Flash Image (nano-banana) | Yes | | Google | Gemini 3 Pro Image (preview) | Yes | | Google | Imagen 4 Fast | No | | Google | Imagen 4 | No | | Google | Imagen 4 Ultra | No | | OpenAI | DALL-E 3 | No | | OpenAI | GPT Image 1 | No | | OpenAI | GPT Image 1 Mini | No | | xAI | Grok 2 Image | No | Picking the right model is a tradeoff between cost, latency, and quality: - **Fastest / cheapest**: Imagen 4 Fast, GPT Image 1 Mini. - **Best general quality**: Imagen 4 Ultra, GPT Image 1, Gemini 3 Pro. - **Need reference image**: any Gemini model. If your org has multiple image models configured, the model picker appears in the AI tab. ## Reverting an image change To restore the original image: 1. Open the **Changes** panel at the bottom of the side panel. 2. Find the row corresponding to the image change. 3. Click **Revert**. Revert clears the image swap _and_ any `srcset` / ` > source[srcset]` removals the editor added when applying it. The element returns to exactly its pre-variation state. ## Storage and privacy Uploaded and AI-generated images are stored in GrowthBook's image storage bucket, namespaced by your organization ID. They're served from a CDN-fronted URL so SDKs can fetch them efficiently when delivering variations. A few specifics: - **Permanent storage** (accepted into a variation) lives at `/visual-editor/`. - **Quarantine storage** (AI candidates not yet accepted) lives at `gen//visual-editor/` and is auto-cleared after 7 days. - **Cross-org isolation** — the library API only returns objects under your org's prefix. Other organizations' images are never visible. - **Image URLs are public** in the sense that anyone with the URL can fetch the image. Don't AI-generate images containing sensitive information. For AI generation specifically, the prompt text and (if used) the reference image are sent to the configured AI provider — see [AI Integrations](/integrations/ai) for what each provider does with that data. ## Image-based experiment ideas A few common patterns that move metrics: - **Hero photography style**: studio vs. lifestyle, person vs. product. - **Product framing**: white background vs. in-use, single product vs. context. - **Illustration vs. photography** for explainer sections. - **Color palette match**: does an image that matches your brand palette outperform a contrasting one? - **Faces vs. objects**: images with human faces often drive higher engagement, but the effect is category-specific. Always pair an image change with the right success metric (CTR, conversion, time-on-page) — image impact varies enormously by industry and audience. ## Next steps - **[AI mode](/app/visual/ai-mode)** — generate images as part of a larger AI-driven variation. - **[Manual mode](/app/visual/manual-mode)** — full reference for the rest of the structured editor. - **[Preview and QA](/app/visual/preview-and-qa)** — preview the variation before publishing. - **[AI Integrations](/integrations/ai)** — provider setup and the full AI feature list. --- ## Install and Connect the GrowthBook Visual Editor import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; The GrowthBook Visual Editor is a standalone Chrome extension. Setup is two clicks and about thirty seconds — install the extension, then click **Connect with GrowthBook**. ## 1. Install the extension Install **[GrowthBook Visual Editor](https://chromewebstore.google.com/detail/growthbook-visual-editor/nbomejknbpkcpjdagefhichaajpoempk)** from the Chrome Web Store (Chrome, Edge, Brave, Dia, and most Chromium browsers). {/* Is firefox support coming soon? */} {/* Firefox support coming soon. */} ## 2. Open the side panel ![The GrowthBook Visual Editor Getting Started](/images/visual-editor/ve-connect-step1.png) Open the extension menu (top right) and choose **GrowthBook Visual Editor**. The first time you open the side panel, you'll see a **Connect with GrowthBook** button. ## 3. Click "Connect with GrowthBook" Click **Connect with GrowthBook**. A new tab opens to GrowthBook: 1. **Already signed in, one org** — a Visual Editor access key is generated and passed back to the extension automatically. The tab closes; the side panel shows you're connected. 2. **Not signed in** — log in first, then the flow continues as above. 3. **Multiple orgs** — pick which org to connect to from a dropdown, then the key is generated. No copying tokens or pasting strings. ### Where the key is stored The access key is stored in the extension's local storage on your machine. It's not synced across browsers, and it's only sent in the `Authorization` header of requests to the GrowthBook API host the extension is configured for. ### Self-hosted GrowthBook :::tip Self-hosted GrowthBook Visual Editor support is coming soon. ::: The one-click flow works on GrowthBook Cloud and self-hosted instances the extension has synced with before. On a fresh install pointing at self-hosted: 1. Open the gear menu → **API host** and set your host (e.g. `https://api.growthbook.io` or `https://localhost:3100`). 2. Paste a [personal access token](https://app.growthbook.io/account/personal-access-tokens) (or the equivalent on your instance). 3. Click **Connect with GrowthBook**. Still failing? See [Troubleshooting → Extension can't connect](/app/visual/troubleshooting#extension-cant-connect-to-my-self-hosted-growthbook). ## 4. The gear menu Once connected, there's a gear icon in the side panel header. Open it for: - **Status** — shows whether the extension is connected and whether the editor is targeting the active tab's domain. - **Language** — change the side panel's language. Independent of your GrowthBook web UI language. - **Dark mode** — toggles the side panel's UI theme. Independent of your GrowthBook web UI theme. - **Change Connection** — change which GrowthBook host you're talking to (e.g., switch from production to staging, or set a self-hosted host) or change the personal access token. - **Disconnect** — clear the stored key and return to the disconnected state. Use this when handing off the browser to someone else. ## Next steps - **[AI mode](/app/visual/ai-mode)** — write your first variation. - **[Running on your site](/app/visual/running-on-your-site)** — install the SDK so variations render in production. --- ## Visual Editor (Legacy) — In-Page Overlay import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; :::note You're reading the legacy docs This page documents the **original in-page overlay** Visual Editor. The current Visual Editor is a Chrome side panel — see [Visual Editor](/app/visual) for the up-to-date docs. These docs are kept as a reference for users still on older versions of the GrowthBook DevTools extension, and for the support team. ::: With the Visual Editor, users can design A/B tests on their site directly in their browser, run them in production, and analyze results, all without writing a single line of code. To use the Visual Editor, a software developer will need to integrate GrowthBook's [Script Tag](/lib/script-tag), [JavaScript](/lib/js), or [ReactJS](/lib/react) SDKs with your application. Alternatively, use one of our [Edge SDKs](/lib/edge/cloudflare) to render visual experiments on a CDN. :::warning Visual Editor may not work optimally on client-side rendered apps With client-side rendered apps (e.g., React, Vue, Svelte), the Visual Editor may not work optimally. Consider using [Feature Flags](/app/features) instead for a better experience. ::: ## Requirements All you need to get started is the [GrowthBook DevTools browser extension](/tools/chrome-extension) installed on your [Chrome](https://chrome.google.com/webstore/detail/growthbook-devtools/opemhndcehfgipokneipaafbglcecjia) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/growthbook-devtools/) browser. Once you've added it to your browser, you need to add a Personal Access Token (GrowthBook menu). The application you want to experiment on **must be a front-end web application viewed in a browser**. Our Visual Editor does not work for mobile apps (Native or ReactNative) or desktop apps (e.g. Electron). For unsupported platforms, we recommend using [Feature Flags](/app/features) instead to implement your changes in code. ### Allow Visual Experiments on the SDK Connection Ensure that the "Include Visual Experiments" toggle is enabled in the SDK Connection's settings, otherwise visual experiments will not be sent to your application. To check if visual experiments are enabled for your SDK connection in GrowthBook: 1. Go to the Left navigation menu > SDK Configuration > SDK Connections page in the GrowthBook UI. 2. Select the SDK connection in question. 3. Click the purple "Edit" button in the upper right corner. 4. Scroll down and look for the "Include Visual Experiments" toggle. - If this toggle is enabled, visual experiments are included in the SDK payload and are enabled for the SDK. - If it is disabled, visual experiments will not run and will not be sent to the application. If you have enabled the toggle and still do not see visual experiments, double-check that you are using the correct client key and SDK connection in your application. ## Creating a Visual Experiment To use the visual editor, first add a new Experiment. This can be done under **Experiments** in the left nav. Select the option to design a new experiment. Then, you'll have a series of fields to fill out (hypothesis, variation names, goal metrics, etc.). Don't worry, these can all be changed later. Once you created an experiment, you should be prompted to open the Visual Editor. Make the desired visual changes using the editor. Once you've created your first experiment and are ready to deploy to production, there are some additional steps required (see **Deploying to Production** below). ### URL Targeting GrowthBook needs to know what page(s) of your site the experiment should run on. :::note URL Targeting is required for the Visual Editor to work For URL Targeting to work, you must pass the `url` targeting attribute into the GrowthBook SDK and also list it in the GrowthBook App. See [Targeting Attributes](/features/targeting) to learn more about targeting users with certain attributes. ::: If your experiment is going to be on a single static page, enter the full URL and submit (e.g. `https://www.example.com/pricing`). If your experiment is going to be on a page with a dynamic URL (e.g. all pages that start with `/post/`), click on the "Advanced Mode" link. You'll need to enter 2 different URLs. 1. A single representative URL you want to load in the Visual Editor (e.g. `https://www.example.com/post/my-first-post`) 2. A URL targeting pattern to match all of your dynamic URLs The targeting pattern supports wildcards (`*`), so for this example, you could enter `/post/*`. :::tip Simple URL targeting is recommended We recommend sticking with "Simple" URL targeting rules (don't be fooled by the name, it's actually really powerful). The other option (Regex) can be useful for really advanced use cases, but it is much harder to write and more error prone. ::: #### Simple Targeting Our "simple" URL targeting option supports the vast majority of use cases and is easy to use. It supports the following features: - Match based on full URLs (e.g. `https://www.example.com/pricing`) - Match based on path (e.g. `/pricing`) - Match based on query strings (e.g. `/pricing?utm_source=email`) - Match based on hashes/anchors (e.g. `/pricing#more-info`) - Wildcards (e.g. `/posts/*` will match `/posts/123` AND `/posts/2023/03/30/my-post`) - Ignores leading and trailing slashes (e.g. `pricing`, `/pricing`, and `/pricing/` are identical) - Ignores the protocol (e.g. `https://...` will also match `http://...`) - Ignores extra query string parameters (e.g. `/pricing/?plan=pro` will match `/pricing/?utm_source=email&plan=pro&logged-in=true`) #### Regex Targeting Our "regex" URL targeting option supports full regular expressions. Writing regular expressions for URLs is very error prone, so be careful and make sure you escape all special characters you don't want to be interpreted. Here's a full example: ```js https?:\/\/(www\.)?example\.com\/pricing\/? ``` You can also match on just the path: ```js ^\/pricing\/(pro|enterprise) ``` ### The Visual Editor There are a number of different tools in the Visual Editor. ![The Visual Editor UI](/images/visual-editor-ui.png) At the top is a dropdown where you can select which variation you are currently editing. Below that is your toolbar. It has the following tools in order from left to right: 1. **Interactive Mode** - Click around your site normally 2. **Selection Mode** - Point and click to select an element on your site to edit. This is the most common way to make changes. 3. **Global CSS** - Inject global CSS styles into the page. Use this to control things like page background color or font size. 4. **Custom JavaScript** - Inject Javascript into the page. Use this to create complex variations. 5. **Change List** - See a summary of all of the changes you've made to the page so far When you're using the Element Selector, after you pick an element to edit, you'll be able to modify the Inner HTML (i.e. the copy), any attributes (e.g. a link HREF), and the list of CSS classes. When you're finished making changes, click the **Done Editing** button to be taken back to GrowthBook. ### Custom JavaScript Custom JavaScript is executed as quickly as possible, often times before the page has fully loaded. This gives you the most flexibility in how to implement your experiment. If you are making changes to elements on the page, make sure you wait until they exist. Below is a small helper function you can add to the top of your Custom JavaScript to help with this: ```js function waitFor(selector) { return new Promise(resolve => { const el = document.querySelector(selector); if (el) return resolve(el); const observer = new MutationObserver(() => { const el = document.querySelector(selector); if (el) { observer.disconnect(); resolve(el)} }); observer.observe(document, {childList: true, subtree: true}); }); } ``` Then, you can use it like so: ```js waitFor(".my-element").then((el) => { el.innerHTML = "Hello World!"; }); ``` ### Drag and Drop When in Selection Mode, you can select an element and click to **drag and drop** it positionally on the page. #### Drag and Drop Handle When an element is selected, you will see a floating handle to move it positionally. Alternatively, you can click anywhere on the selected element to move it as well. ![Visual Editor Drag and Drop Handle](/images/visual-editor-move-handle.png) Once dragging, the cursor will highlight an edge where the dragged element will land. When you release the cursor the element will move to the indicated spot. If you want to undo the move, click the 'Undo' button that is available for elements that have been dragged. ![Visual Editor Drag and Drop Example](/images/visual-editor-drag-and-drop.gif) ### Debug Panel The Visual Editor now comes with a Debug Panel to help diagnose any issues with your SDK configuration and URL Targeting rules. Use this to help diagnose your own errors, or provide a screenshot to our Support team when reaching out to help provide context. ![Debug Panel](/images/visual-editor-debug-panel.png) ## Deploying to Production Before you start your first experiment, you will first need to generate a **Client Key** in GrowthBook. You can get this by creating a new SDK Connection (under **SDK Configuration** in the left navigation). **Important**: Make sure to enable the "Include Visual Experiments" toggle. If you forget this step, Visual Editor experiments will not be sent to your application. ![Enable the Visual Editor in your SDK Connection](/images/sdk-connection-visual-editor.png) You can also optionally include Draft Experiments (see below). Once you create an SDK Connection, you need to follow the steps to integrate GrowthBook into your website. The easiest and recommended way to do this is with our [Script Tag SDK](/lib/script-tag), which works out-of-the-box and doesn't require any configuration for most websites. It's also possible to use either our [JavaScript](/lib/js) or [ReactJS](/lib/react) SDKs, although these do require more up-front work to integrate into your application. :::note Updating your SDK configuration If you've already been using our front-end SDKs for feature flagging, make sure you: 1. Update to the latest SDK version 2. Follow the "Visual Editor" instructions in the SDK docs ::: ### Content Security Policy (CSP) Changes If your website uses a Content Security Policy (CSP), there are some additional changes you'll need to make. This applies to both SDK and pre-build script tag integration. #### script-src Changing the `script-src` directive in your CSP is only required if you are writing custom JavaScript in the Visual Editor. If you are only changing styles or copy using the point-and-click editor, this is not required and you can skip this section. If you have the `script-src` directive defined in your website's CSP, you'll need to enable `'unsafe-inline'` and `'unsafe-eval'` in order to leverage the Global JavaScript injection feature of the Visual Editor: ``` Content-Security-Policy: script-src 'self' 'unsafe-inline' 'unsafe-eval'; ``` ##### Using Script Nonces As an alternative to allowing `unsafe-inline`, we support "nonces", although this requires some very technical and custom configuration to hook up. First, you will need to generate a unique nonce value for every request and add it to your CSP header. This can be done on the edge such as with a Cloudflare Worker. Then, you will need to pass this nonce into your GrowthBook SDK as `jsInjectionNonce`. For example, if you are using our [Script Tag SDK](/lib/script-tag), you can add the following into your page's `` BEFORE you load the GrowthBook snippet. Replace all instances of `$NONCE` with the unique nonce value you generated. ```html window.growthbook_config = window.growthbook_config || {}; window.growthbook_config.jsInjectionNonce = "$NONCE"; ``` You will still need to allow `unsafe-eval`. ## Drafts and QA While the experiment is still a draft, you can preview variations by adding a querystring to your URL. :::note Drafts require turning on the "Include Drafts" toggle for your SDK Connection in GrowthBook. ::: To build the QA preview URL, you'll need the **Experiment Id** (viewable on the right side of the experiment page under Settings). You'll also need the variation number you want to preview. `0` is the control, `1` is the 1st variation, etc.. Now, just join these together with an equals sign (e.g. `my-experiment-id=1`). This needs to go in the Querystring part of the URL (after a question mark). Here's a full example: `https://www.example.com/pricing?my-experiment-id=1` Until an experiment is moved out of the "draft" phase and started, this is the only way to view it on your site. ## Stopping an Experiment When your experiment is finished, you can click on the `Stop Experiment` link at the top of results. This will prompt you for several bits of information about why you're stopping and what the conclusion was. If your variation won, you can optionally enable a `Temporary Rollout` when stopping. This will continue running your experiment with the same targeting conditions, but send 100% of traffic to the winning variation and disable the `trackingCallback` from being called. The reason it's called a "Temporary" Rollout is because you don't want to rely on our SDK to implement the winning variation forever. It's best practice to have your engineering team re-implement the changes directly in your site's code. This is for a number of reasons: 1. Changes implemented in code are rendered quicker, so your site will load faster 2. Changes in code will be picked up for SEO 3. Changes applied through the visual editor require the SDK to download data from GrowthBook. Although lightweight, these stopped experiments can add up over time and further slow down your site. 4. Reduce the chance of conflicts. If two visual editor experiments try to change the same element at the same time, it will not always work as expected. Moving the winning variation to code will avoid this issue. --- ## Manual Mode — Point-and-Click A/B Test Editor import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; Manual mode is the point-and-click editor inside the [Visual Editor side panel](/app/visual). Click an element on your page, and you get structured controls for everything you can change on it — content, typography, background and border, spacing, and visibility — plus an escape hatch for arbitrary CSS and JavaScript. Manual mode is the right choice when: - You know exactly what you want to change and you want pixel-level control. - You're fine-tuning a variation that started in [AI mode](/app/visual/ai-mode). - The change is too narrow to be worth a prompt (e.g., "make this padding 12px"). - You want to copy a specific style from a design spec. ## Picking an element Switch to the **Manual** tab in the side panel. With the panel open, hover any element on your page — it highlights with a blue overlay. Click to select it. A sticky bar at the top of the side panel shows what's currently selected (its tag, classes, and a short selector path). You can: - **Pick a different element** — click another element on the page, or click the picker icon in the sticky bar. - **Navigate nearby** — use the "tree" icon to navigate to nearby elements. See below - **Deselect** — click the x on the right side of the selected bar or pick another element to deselect the current one. ### Detailed Selector ![The GrowthBook Visual Editor detailed selector](/images/visual-editor/ve-detailed-selector.png) Sometimes clicking on an element selects a parent or child instead of the one you want. In that case, use the sticky bar's selector path to select the exact element. The path shows the current element and its parent elements, sibling elements, and children. Hover over the path to see it selected on the page. ## Property groups Once you've selected an element, the panel fills with collapsible property groups. Each group corresponds to a category of change. ### Content ![The GrowthBook Visual Editor HTML options](/images/visual-editor/ve-text-editor.png) ![The GrowthBook Visual Editor HTML editor](/images/visual-editor/ve-text-editor-html.png) Edit what the element actually says and links to: - **Inner HTML** — change the visible text or markup inside the element. Multiline-supported. If the selected element has HTML inside of it, you'll be presented with two modes for editing content: edit as text and flatten the HTML, or edit as HTML. ### Links ![The GrowthBook Visual Editor Link Editor](/images/visual-editor/ve-link-editor.png) If the selected element is a link (``), you'll get additional controls for editing the URL and link behavior. It will let you set the `href`, and the option to open in a new tab (`target="_blank"`). If the link opens in a new tab, it will add `rel="noopener noreferrer"` attribute for security. ### Classes ![The GrowthBook Visual Editor Class Editor](/images/visual-editor/ve-class-editor.png) The **Classes** field shows the element's current classes and allows you to add or remove classes. This is a quick way to apply existing styles from your CSS without needing to copy individual properties. ### Typography ![The GrowthBook Visual Editor Typography editor](/images/visual-editor/ve-typeography.png) Controls for text styling: - **Font size** — with a unit selector (`px`, `em`, `rem`, `%`) - **Font weight** — preset weights and custom numeric input - **Color** — color picker plus hex input - **Alignment** — left / center / right / justify (segmented control) - **Line height** — unitless, `px`, `em`, `rem`, or `%`. Empty unit (`-`) means unitless, which is the CSS recommendation for line-height. - **Letter spacing** — `px`, `em` - **Text decoration** — none / underline / strikethrough / overline Alignment and text decoration share a single row to keep the panel compact. ### Background & Border ![The GrowthBook Visual Editor Background and Border](/images/visual-editor/ve-background-borders.png) For containers, cards, and any visually styled element: - **Background color** — color picker with a clear option - **Border** — width, style (solid / dashed / dotted / none), and color. The color swatch is positioned so the native color picker doesn't render off-screen for elements near the right edge of the viewport. - **Box shadow** — None / Small / Medium / Large presets. The presets follow common design-system scales; for custom shadows, use Global CSS. - **Background image** — paste a URL or use one of the [image flows](/app/visual/images). Hidden on `` elements (use Content → image replace instead). #### Background Images ![The GrowthBook Visual Editor Background Images](/images/visual-editor/ve-background-image.png) The background image area allows you to set a background image on any element. This is similar to the `` replacement, but has a few more options for fitting that background images have. ### Layout ![The GrowthBook Visual Editor Layout](/images/visual-editor/ve-layout.png) The Layout group includes controls for choosing the display mode and adjusting spacing with padding and margin editors. - **Display type** — choose between `block`, `inline-block`, `inline`, `flex`, `grid`, and `none`. Setting to `none` is a quick way to hide an element without removing it from the DOM. - **spacing** — click and drag horizontally over a value to scrub it up or down (similar to Figma) for padding and margin. - **Width & Height** — Select the width, max-width, height and max-height for your element. With unit selectors. Setting to `auto` removes the mutation. Manual mode writes per-side longhands (`padding-top`, `margin-right`, etc.) and clears any shorthand `padding`/`margin` declaration, so the CSS cascade behaves predictably. ### Visibility - **Hide this element** — a quick toggle that sets `display: none`. - **Opacity** — slider from 0% to 100% plus a number input. Setting to 100% removes the mutation (no point keeping `opacity: 1` on the element). ## Global CSS and Custom JavaScript For changes that don't fit cleanly into a single element's property groups, use the top-level **CSS** and **JS** tabs: ### Global CSS Inject a CSS block that runs alongside the variation. Useful for: - Page-wide background color or font changes - Hover states that the structured controls can't express - Animations - `@media` queries for responsive tweaks (the only way to scope changes per breakpoint today) The CSS is inserted into a `` tag scoped to the variation and removed when the user is bucketed back to control. ### Custom JavaScript Inject a JavaScript block. Use sparingly — it's an escape hatch for changes that genuinely need imperative code (e.g., DOM nodes that don't exist until the user clicks something). Custom JS runs as early as possible, often before the page has finished loading. If you're targeting elements that load late, wait for them: ```js function waitFor(selector) { return new Promise(resolve => { const el = document.querySelector(selector); if (el) return resolve(el); const observer = new MutationObserver(() => { const el = document.querySelector(selector); if (el) { observer.disconnect(); resolve(el); } }); observer.observe(document, { childList: true, subtree: true }); }); } waitFor(".my-element").then((el) => { el.innerHTML = "Hello World!"; }); ``` Custom JS requires extra CSP changes on your site — see [Content Security Policy](/app/visual/running-on-your-site#content-security-policy-csp) on the Running on your site page. ## Next steps - **[AI mode](/app/visual/ai-mode)** — draft variations from a prompt, then fine-tune here. - **[Images](/app/visual/images)** — upload, AI-generate, and reuse images across experiments. - **[Preview and QA](/app/visual/preview-and-qa)** — preview links, debug panel, live-experiment warning. - **[Troubleshooting](/app/visual/troubleshooting)** — when a variation isn't behaving as expected. --- ## Preview and QA Visual Editor Experiments import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; QA is where good A/B tests are made — or where bad ones get caught before they're launched on real users. The [Visual Editor](/app/visual) gives you several tools to verify a variation works on a real browser, on the real page, with the real SDK, before publishing. ## Preview links While an experiment is in draft, you can preview any variation by adding a querystring to your URL: ``` https://www.example.com/pricing?my-experiment-id=1 ``` Where: - `my-experiment-id` is the experiment ID (visible on the right side of the experiment page in GrowthBook under **Settings**). - The value is the variation index — `0` for control, `1` for the first variation, `2` for the second, and so on. The SDK on your site sees the querystring, bypasses its usual targeting/bucketing logic, and forces the user into the named variation. This makes preview links shareable: send the link to a teammate or stakeholder and they'll see the variation on their own browser. ### Enable "Include Drafts" on the SDK Connection Preview links only work if the SDK has the experiment definition. Draft experiments are _not_ included in the SDK payload by default — you have to opt in: 1. In the GrowthBook UI, go to **SDK Configuration → SDK Connections**. 2. Open the connection your site uses. 3. Click **Edit** and enable the **Include Drafts** toggle. :::warning Include Drafts exposes all drafts Turning on Include Drafts means _every_ draft experiment is in your SDK payload, not just the one you're previewing. This is fine for staging environments but generally not what you want in production. We recommend either using a separate SDK Connection for QA traffic, or only enabling Include Drafts temporarily. ::: ### Once the experiment starts After the experiment is moved out of draft into the running state, the preview-link querystring still works the same way and `?my-experiment-id=1` still forces variation 1. The difference is that real users (without the querystring) now get bucketed normally. This is helpful for ongoing QA — if a stakeholder reports "the variation looks wrong," you can send them the preview link to confirm whether the variation is in fact wrong (vs. the user just got bucketed differently than expected). ### Why preview links sometimes don't work The most common causes: - **Include Visual Experiments is off** on the SDK Connection. Preview links can't deliver a variation the SDK doesn't have. See [the landing page](/app/visual#allow-visual-experiments-on-the-sdk-connection). - **Include Drafts is off** and the experiment is still a draft. The SDK doesn't have the experiment yet. - **Wrong SDK Connection / client key**. Your site is loading a different SDK Connection than the one the experiment is associated with. - **The URL doesn't match the URL targeting rule.** Even with the querystring, the SDK won't run the experiment on URLs that don't match. Double-check the targeting pattern. - **Caching at the CDN or browser level.** A previous request without the querystring may have been cached. Hard-reload (Cmd-Shift-R / Ctrl-Shift-F5) to bypass. ## The debug panel The Visual Editor includes a Debug Panel showing what the SDK actually evaluated on the active page. Use it to diagnose why a variation isn't showing. The panel surfaces: - **SDK initialization** — did the SDK load? Which client key is it using? - **URL targeting** — what URL did the SDK see, and which experiments matched? - **Bucketing** — which variation did the SDK assign for each running experiment? - **Visual changes** — which DOM mutations did the SDK apply, on which elements? If you're filing a support ticket about a variation not rendering, including a screenshot of the Debug Panel saves several rounds of back-and-forth. ## The live-experiment editing warning When you open the Visual Editor against an experiment that's **already running**, the side panel shows an amber warning banner at the top. Here's why that matters: - A running experiment is actively bucketing users. - Visual variations are served from the SDK Connection payload, which the SDK refreshes periodically. - When you save a change to a running experiment's variation, the new variation is immediately visible to every bucketed user, the next time their SDK refreshes — typically within minutes. This is sometimes useful (you spotted a typo in a live variation and want to fix it without restarting). It's more often a foot-gun (you wanted to draft v2 of the variation but accidentally edited v1, and now users are seeing your half-finished work). If you want to iterate on changes without affecting live users, create a **new variation** or a **new draft experiment** instead. The amber banner can be dismissed for the current session but reappears the next time you open the side panel against a running experiment. ## The wrong-domain banner The side panel targets a specific URL (the one you set when creating the experiment). If you switch the active tab to a different domain, the panel shows a banner reminding you that changes won't apply on this page. You have two options: - **Create a new experiment** scoped to the new domain. - **Update the existing experiment's target URL** to match (if you genuinely wanted to switch domains). Visual changes are not applied while the banner is showing, even if you make edits — this prevents the editor from accidentally modifying an unrelated page just because it was in your active tab. ## QA checklist before going live Before moving a Visual Editor experiment out of draft: - [ ] **Test the preview link** for each variation on the real production-like page. - [ ] **Test in multiple browsers** — at minimum Chrome and Safari. Variations sometimes render differently in Safari due to vendor-prefix or layout differences. - [ ] **Test on mobile** — open the preview link on a real phone or use Chrome DevTools' device emulation. - [ ] **Check the Debug Panel** for unexpected matches or mismatches. - [ ] **Verify the SDK Connection** has both Include Visual Experiments and (during testing) Include Drafts turned on. - [ ] **Confirm the success metric** is being tracked on the affected page (check the GrowthBook events stream or your own analytics). - [ ] **Run a quick QA pass with the team** — share the preview link in Slack or your channel of choice. ## Next steps - **[Troubleshooting](/app/visual/troubleshooting)** — when the preview doesn't behave as expected. - **[Visual Editor overview](/app/visual)** — back to the main page. --- ## Run Visual Experiments on Your Site — SDK Setup and Anti-Flicker import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; The [Visual Editor](/app/visual) is where you design A/B tests with a WYSIWYG editor. To actually render those tests on your live site for real users, you need to install one of GrowthBook's SDKs. This page walks through which SDK to pick, how to install it, and how to avoid the most common pitfall of client-side A/B testing — **flicker**, also known as **flash of original content (FOOC)**. ## Create an SDK Connection In the GrowthBook App: 1. Go to **SDK Configuration → SDK Connections**. 2. Click **Add SDK Connection**. 3. Give it a name (e.g., `Production Web`), pick the environment, and choose the SDK language. 4. **Enable the `Include Visual Experiments` toggle.** This is the single most important step — without it, the SDK payload won't include your visual experiments and they will not render. 5. Optionally enable **Include Drafts** if you want preview links to work for draft experiments. We recommend leaving this off in production. ![Enable Include Visual Experiments on your SDK Connection](/images/sdk-connection-visual-editor.png) After saving, GrowthBook gives you code to add to your product. ## Choosing an SDK GrowthBook has four SDKs that can render visual experiments. Pick based on how your site is built and how much you care about flicker: | SDK | Best for | Flicker | Effort | | ------------------------------------ | -------------------------------------------------------------------- | ---------------------------------------------------------------- | ---------------------------- | | **[Edge SDK](/lib/edge/cloudflare)** | Sites served through a CDN edge (Cloudflare, Fastly, Akamai, Vercel) | **None** — variations applied before HTML reaches the browser | Moderate (edge worker setup) | | **[Script Tag](/lib/script-tag)** | Most sites — fastest path to "it works" | Minimal if loaded in ``; add anti-flicker script if needed | Low (one `` tag) | | **[JavaScript](/lib/js)** | Existing JS apps that already use GrowthBook for feature flags | Same as Script Tag | Medium (programmatic init) | | **[React](/lib/react)** | React apps that already use GrowthBook | Same as Script Tag | Medium (provider setup) | **Our strong recommendation:** if you have access to a CDN edge runtime, use the **[Edge SDK](/lib/edge/cloudflare)**. It eliminates flicker entirely and produces a far better experience for users. The Edge SDK is the only option that delivers visual variations as part of the initial HTML response — every other option has to apply changes in JavaScript after the page has begun rendering. If an Edge SDK isn't practical, the **[Script Tag SDK](/lib/script-tag)** in `` is the next best thing. ## Avoiding flicker **Flicker** (or **FOOC** — _flash of original content_) happens when the browser paints the control of a page before the SDK has a chance to apply the variation. The user sees the original page briefly, then the variation snaps in. It looks bad, it can skew engagement metrics, and on slow connections it can be jarring. There are three established strategies. Use as many as apply to your stack: ### Strategy 1: Use an Edge SDK (recommended) The cleanest fix is to apply variations at the edge, before the HTML ever reaches the browser. With an [Edge SDK](/lib/edge/cloudflare), the user receives HTML that's already been mutated to their assigned variation. There's nothing to flicker — the control is never rendered. This is genuinely a category better than the other options. Every client-side approach has _some_ irreducible delay between the browser starting to render and the SDK applying mutations. Edge rendering eliminates that delay entirely, and as a bonus your variations are visible to SEO crawlers that don't execute JavaScript. If you can use the Edge SDK, use it. ### Strategy 2: Load GrowthBook in ``, as early as possible If you can't use an Edge SDK, the next best thing is to make sure GrowthBook is one of the very first things the browser fetches. Put the SDK `` tag at the top of `` — above other scripts, above analytics, above heavy third-party tags. ```html ``` This minimizes the window between page render start and SDK initialization. On a fast connection, the delay can be small enough that flicker isn't perceptible. Avoid: - Putting GrowthBook at the bottom of `` — the page is already painting by the time the script runs. - Loading GrowthBook through a tag manager (GTM, Tealium) — adds an extra round-trip and a noticeable delay. - Heavy synchronous scripts ahead of GrowthBook in `` — they block the SDK from running. ### Strategy 3: Anti-flicker script If you can't use an Edge SDK and head-loading still isn't fast enough (typical on slow mobile networks or pages with lots of synchronous third-party scripts), add an **anti-flicker script**. An anti-flicker script blanks the page (usually by setting `body { opacity: 0 }` or `display: none`) until the SDK has applied variations, then reveals the page. The user sees a brief blank screen instead of the control followed by the variation. This is the same approach **every** client-side A/B testing tool uses. It's a tradeoff: you trade flicker (which looks bad) for a brief blank flash (which makes the page feel slower to appear). GrowthBook's works hard to keep the library size as small as possible and our performance as high as possible. As a result, our experiments load faster than our competition, with or without the anti-flicker scripts. The anti-flicker is enabled through SDK config — see the [Script Tag SDK docs](/lib/script-tag#anti-flicker) for the exact snippet. - **Set a max-wait timeout.** If the SDK fails to load (network error, API outage), you don't want to leave the page blank forever. A typical timeout is 1–4 seconds. ### Strategy 4: Consider whether visual testing is the right tool Heavily client-side-rendered apps (complex React/Vue/Svelte that re-render the DOM after hydration) are especially prone to flicker, because the visual editor's DOM mutations can be overwritten by the framework's re-render. For those, [Feature Flags](/app/features) implemented in code are usually a better fit than visual experiments — the variation is rendered by your application itself, so there's no race condition. ## Content Security Policy (CSP) If your site uses a Content Security Policy, you may need a few additions. ### `script-src` Required **only** if you use the Custom JavaScript injection feature of the [Manual mode](/app/visual/manual-mode#custom-javascript) editor. Pure-style and copy changes don't need it. If you need it, allow `'unsafe-inline'` and `'unsafe-eval'`: ``` Content-Security-Policy: script-src 'self' 'unsafe-inline' 'unsafe-eval'; ``` ### Using script nonces As an alternative to `'unsafe-inline'`, GrowthBook supports nonces. This requires custom edge configuration. First, generate a unique nonce on every request and add it to your CSP header (e.g., in a Cloudflare Worker). Then, pass the nonce to the SDK as `jsInjectionNonce`. With the [Script Tag SDK](/lib/script-tag), set it before loading the GrowthBook snippet: ```html window.growthbook_config = window.growthbook_config || {}; window.growthbook_config.jsInjectionNonce = "$NONCE"; ``` You will still need `'unsafe-eval'`. ## A note on temporary rollouts When an experiment finishes and you enable a Temporary Rollout to ship the winning variation, the variation continues to be delivered through the same SDK Connection. The same flicker considerations apply. Temporary Rollouts are explicitly named **temporary** because long-term you should re-implement the winning variation in your site's code. ## Next steps - **[Preview and QA](/app/visual/preview-and-qa)** — preview links, debug panel, live-experiment warning. - **[Troubleshooting](/app/visual/troubleshooting)** — fixes for common issues, including SDK and flicker problems. - **[Script Tag SDK](/lib/script-tag)** — the recommended quick-start SDK. - **[Edge SDK](/lib/edge/cloudflare)** — recommended for zero-flicker rendering. --- ## Visual Editor Troubleshooting import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; When the [Visual Editor](/app/visual) doesn't behave as expected, this page lists the most common causes and fixes. Each item is independently linkable — share the anchor in support threads. If your issue isn't here, the **Debug Panel** in the Visual Editor (see [Preview and QA](/app/visual/preview-and-qa#the-debug-panel)) is the single best diagnostic tool. Open it before filing a support ticket. ## Variation isn't showing on my site By far the most common issue. The fix is almost always one of: 1. **Refresh the page** Sometimes the visual editor's changes don't apply until a hard reload (Cmd-Shift-R) to bust the cache, especially if you had the page open while making changes in GrowthBook. 2. **`Include Visual Experiments` toggle is off** on the SDK Connection. Visual experiments are stripped from the SDK payload unless this is enabled. Go to **SDK Configuration → SDK Connections**, edit the connection your site uses, and turn it on. 3. **The SDK isn't loaded yet** when the variation tries to render. Custom JS variations need to wait for the target element — use the `waitFor` helper from [Manual mode → Custom JavaScript](/app/visual/manual-mode#custom-javascript). 4. **URL targeting doesn't match.** The current URL doesn't satisfy the experiment's URL targeting rule. Check the rule and verify with the Debug Panel. 5. **The experiment is still a draft** and the SDK Connection doesn't have **Include Drafts** enabled. Draft experiments aren't delivered by default. ## Preview link returns the control You're hitting `?experiment-id=1` but seeing variation 0. In order of likelihood: 1. **The experiment is a draft** and **Include Drafts is off** on the SDK Connection. Toggle it on, then hard-reload (Cmd-Shift-R) to bust the SDK cache. 2. **Wrong client key.** Your site is loading a different SDK Connection than the one the experiment is associated with. Check the SDK Connection in GrowthBook against the client key in your site's source. 3. **CDN or browser cache** is serving the pre-querystring version of the page. Hard-reload, or test in a private window. 4. **URL targeting excludes this URL.** Even with the querystring, the SDK won't run the experiment on URLs that don't match the targeting pattern. ## Element selector picks the wrong thing When you hover an element, the blue highlight selects an ancestor or sibling instead of what you wanted: ![The Visual Editor tree selector, showing the selected element's parents, the current node, its siblings, and children](/images/visual-editor/ve-tree-selector.png) - **Use the tree selector.** Click the **tree icon** in the selected-element bar to open a small view of the surrounding DOM, then click the exact ancestor or child you want. This is the most reliable way to land on the right node. See [Picking an element](/app/visual/manual-mode#picking-an-element). - **Cycle through stacked elements.** For elements sitting on top of each other (z-index), the picker grabs the topmost. Hold a modifier key (Alt on macOS, Shift on Windows) while clicking to select the ones underneath. - **Keep interactive elements open.** Dropdowns and hover menus close as soon as you move the mouse to the side panel. Use the **Interactive Mode** toggle to keep them open while you select. - **Override the selector directly.** In the **Advanced** section you can type or refine the CSS selector by hand — useful when no amount of clicking lands on the element, or when you want a more durable selector. See [Detailed Selector](/app/visual/manual-mode#detailed-selector). ## Changes flicker on page load (FOOC) A "flash of original content" happens when the SDK applies the variation a moment after the page renders the control. Mitigations: - **Load the SDK as early as possible.** Place the `` tag in ``, not at the bottom of ``. - **Use the [Script Tag SDK](/lib/script-tag)** — it's optimized to apply variations before paint when possible. - **Avoid heavy `before-render` custom JS.** The SDK has to wait for it. - **For server-rendered sites, consider an [Edge SDK](/lib/edge/cloudflare)** to apply visual changes at the CDN before HTML reaches the browser. This eliminates FOOC entirely. Heavy client-side-rendered apps (complex React/Vue/Svelte that rehydrate the page) are especially prone to flicker. For those, [Feature Flags](/app/features) are usually a better fit than visual experiments. ## The page freezes or becomes unresponsive while editing Some elements aren't plain HTML — they're rendered and **owned by the site's JavaScript framework** (React, Next.js, Vue, Svelte, etc.). When you edit one, the Visual Editor applies your change, the framework re-renders the element and overwrites it, the editor re-applies, the framework reverts again — and the two can spin the browser's main thread until the page becomes unresponsive. The editor detects this loop and steps in before the page freezes. What happens next depends on the change: - **Style changes are fixed automatically.** Color, size, spacing, borders, visibility, and other styling are **re-delivered as global CSS** — a CSS rule re-matches the element on every render instead of being overwritten, so it beats the framework without a fight. These just work (you'll find them in the variation's global CSS); nothing to do. - **Content and structural changes can't be.** Text, HTML, and attribute changes have no CSS equivalent, so the editor **stops previewing** the change (to stay responsive) but **keeps it saved** — the loop can be specific to the editor (dev/staging builds re-render far more than production, and you're interacting with the page while editing), so the change may still work when the SDK applies it once on your live site. When you see the notice: - **Verify with a [preview link](/app/visual/preview-and-qa)** on your live site. The editor can't tell an editor-only loop from a real one, so confirm whether the change actually holds for users before relying on it. - **Target a nearby static element** — one the framework doesn't re-render. Use the [tree selector](#element-selector-picks-the-wrong-thing) to look for a stable parent or child. (An adjacent element isn't guaranteed to be static — much of a modern app's DOM is framework-controlled, so this doesn't always help.) - **Use a [feature flag](/app/features)** instead. When the site's own code renders the content, a code-level change is the right tool — the visual editor is fighting the framework for ownership of that element. - **As an escape hatch, Custom JS** with a `waitFor`/`MutationObserver` helper can re-apply the change, though the framework may still overwrite it. See [Custom JavaScript](/app/visual/manual-mode#custom-javascript). This also affects **live experiments** — the SDK applies changes the same way — so a change that fights the framework can flicker for real users too. For heavily client-rendered pages, prefer styling (delivered as CSS) or [Feature Flags](/app/features) over text/DOM changes. ## Image upload didn't replace the picture Modern responsive images use `srcset` and `` markup. If a plain `src` replacement doesn't show up: - The Visual Editor should be removing `srcset` automatically — check the Changes panel for an attribute-removal change next to the image change. - If it didn't, the target element might be nested in markup the picker didn't recognize as a ``. Try selecting the parent `` element directly. - See [Images → `` and `srcset` handling](/app/visual/images#picture-and-srcset-handling). ## AI mode is greyed out / not available AI mode requires: - **An AI provider key configured** by an org admin in **Settings → AI**. See [AI Integrations](/integrations/ai). - **A plan that includes the `ai-suggestions` commercial feature.** Free plans don't include AI. - **An active connection** in the side panel — sign in if you've been signed out. If all three are in place and AI mode is still unavailable, hard-reload the side panel (right-click → Reload). ## Extension can't connect to my self-hosted GrowthBook - **Confirm the API host URL.** Use your API host, not your front-end host. They're usually different (e.g., `api.example.com` vs. `app.example.com`). - **Confirm HTTPS.** Chrome blocks extension requests to HTTP hosts and to HTTPS hosts with invalid certs. Use a real TLS cert. - **Test the API from `curl` first** with `Authorization: Bearer `. If `curl` fails, the host or token is the problem. If `curl` works but the extension doesn't, your browser's network environment (proxy, VPN, split-DNS) is blocking extension requests. - **CORS** — your API host's CORS configuration must allow the extension's origin. The extension uses `chrome-extension://` as its origin. ## "Wrong domain" banner won't go away The Visual Editor scopes itself to the URL you set when creating the experiment. If you've moved to a different domain in the active tab, changes are paused intentionally. - **Switch the active tab back** to the experiment's target URL. - **Or update the experiment's target URL** in GrowthBook to match where you actually want to edit. ## Changes apply to the wrong variation - **Check the variation switcher** above the property groups. It's easy to make changes to v2 thinking you're on v1. - **The control is read-only** — if you can't make changes, you're probably on the control tab. Switch to one of the variants. ## JavaScript changes seem to apply to every variant This is an **editor-preview artifact** — it doesn't happen for real users. When you add or **Run** Custom JS, the script executes on the live page so you can preview it. CSS and DOM/text changes are cleanly undone when you switch variants, but **arbitrary JavaScript can't be automatically reversed** — once a script has run it may already have created elements, attached listeners, or changed page state. Switching variants in the editor doesn't "un-run" it, so the page shows the **combined effect** of every script you've run since the last page load. **Real users only ever see one variant** — each visitor is bucketed into a single variation and loads only that variation's JS. They never experience the stacking you see while flipping between variants in the editor. To preview one variant cleanly: - **Reload the page**, then open the variant you want. A fresh load runs only that variant's JS. (The editor prompts you to reload after you change a JS block, since JS can't be hot-swapped in place.) - Use **Run** to re-execute the current variant's JS after editing it — Custom JS runs once per page load, not reactively. - To be sure no leftover side effects remain, hard-reload (Cmd/Ctrl-Shift-R). If your JS targets elements that appear late, wrap it in a `waitFor` helper — see [Custom JavaScript](/app/visual/manual-mode#custom-javascript). ## Save fails with "permission denied" Your Personal Access Token's user account doesn't have permission to edit this experiment. Check: - **Project permissions.** Your user role might be limited to a subset of projects, and this experiment lives in a different one. - **Experiment ownership.** Some orgs restrict editing to experiment owners or members of the assigned team. - **PAT scope.** Re-create the PAT if it was generated under an older account state. ## Still stuck? - **Share the Debug Panel screenshot** when filing a support ticket — see [Preview and QA → The debug panel](/app/visual/preview-and-qa#the-debug-panel). - **Reach out** in [the GrowthBook community Slack](https://slack.growthbook.io) or via [support](mailto:support@growthbook.io). ## Next steps - **[Preview and QA](/app/visual/preview-and-qa)** — preview links, debug panel, live-experiment warning. - **[Visual Editor overview](/app/visual)** — back to the main page. --- ## Visual Editor import MaxWidthImage from '@site/src/components/MaxWidthImage'; import CommercialFeature from '@site/src/components/CommercialFeature'; GrowthBook's AI-based Visual Editor is a no-code way to build and launch experiments on your website. Describe the change you want in natural language, review what the editor proposes, and ship a real A/B test without writing code or waiting on a deploy. :::tip Install the extension Get the [GrowthBook Visual Editor](https://chromewebstore.google.com/detail/growthbook-visual-editor/nbomejknbpkcpjdagefhichaajpoempk) from the Chrome Web Store, then [connect it to your account](/app/visual/install-and-connect). ::: The Visual Editor is a **Chrome side-panel extension**. You open it next to your site, pick the page you want to test, and make changes two ways: - **[AI mode](/app/visual/ai-mode)** — describe the change in plain language ("make the hero headline shorter and the CTA green") and the editor proposes the edits for you to preview and accept. - **[Manual mode](/app/visual/manual-mode)** — point and click to select any element and edit its content, typography, layout, background, classes, and more with structured controls. - **[Figma & image import](/app/visual/design-import)** — turn a Figma frame or a mockup image into a high-fidelity, testable component the editor builds and places on the page for you. To actually render your variations on your live site, a developer integrates one of GrowthBook's front-end SDKs — the [Script Tag](/lib/script-tag), [JavaScript](/lib/js), or [ReactJS](/lib/react) SDK — or renders them on a CDN with an [Edge SDK](/lib/edge/cloudflare). See [Running on your site](/app/visual/running-on-your-site). :::note Looking for the old in-page editor? The Visual Editor is now a Chrome side panel. If you're using an older version of the GrowthBook DevTools extension with the in-page overlay editor, see the [Legacy (in-page) editor docs](/app/visual/legacy). ::: ## Benefits - **Prompt-driven editing**: Describe changes to text, images, buttons, spacing, and more in [AI mode](/app/visual/ai-mode). Pin an element when you want the AI to focus on a specific spot on the page. - **Run experiments without dedicated eng time**: Draft and QA variations in the browser; your site still needs the GrowthBook SDK for production delivery. See [Running on your site](/app/visual/running-on-your-site). - **Manual precision**: Fine-tune typography, spacing, links, and visibility with structured controls when you know exactly what you want. See [Manual mode](/app/visual/manual-mode). - **Transparent by design**: Preview links, a debug panel, and warnings when you edit a live experiment. See [Preview and QA](/app/visual/preview-and-qa). :::info Join the Visual Editor community on Slack Connect with the GrowthBook team and other users. [Join the Visual Editor channel on Slack](https://growthbookusers.slack.com/archives/C0BBMTLJUUT). ::: ## Quick start 1. [Install and connect](/app/visual/install-and-connect) the Chrome extension (~30 seconds). 2. Open the side panel on the page you want to test and start editing in [AI mode](/app/visual/ai-mode) or [Manual mode](/app/visual/manual-mode). 3. [Set up the SDK](/app/visual/running-on-your-site) on your site so variations render for real users. Hitting a snag? See [Troubleshooting](/app/visual/troubleshooting). ## Requirements Your app must be a **front-end web app in a browser** (not native mobile or desktop). For other platforms, use [Feature Flags](/app/features) in code. :::warning Client-side rendering can interfere with visual changes Heavily client-side-rendered apps (React, Vue, Svelte) may flicker or fight the SDK when the framework re-renders the DOM. See [Running on your site → Strategy 4](/app/visual/running-on-your-site#strategy-4-consider-whether-visual-testing-is-the-right-tool) for details and alternatives. ::: ### SDK setup Visual experiments only render if your SDK Connection has **Include Visual Experiments** enabled and the correct SDK is installed on your site. See [Running on your site](/app/visual/running-on-your-site#create-an-sdk-connection) for the full setup. --- ## Setting up Athena as a data source import MaxWidthImage from "@site/src/components/MaxWidthImage"; Setting up Athena as your datasource requires you set up the proper permissions within AWS for Growthbook to access and then provide the correct credentials to Growthbook make use of those permissions. There are also optional connection data that will help Growthbook create the correct default sql to analyze your data. ## Setting up Permissions in AWS Unlike other database engines with their own user management system, Athena uses IAM for authentication. ### Growthbook Cloud We recommend creating a new IAM user with readonly permissions for GrowthBook. The managed [AWSQuicksightAthenaAccess](https://docs.aws.amazon.com/athena/latest/ug/managed-policies.html) is a good starting point. You will also need to give it permission to read from the s3 tables that hold your event data, by taking a modified version of `AmazonS3ReadOnlyAccess` policy whose resources are confined to only those tables that hold your event data. For example with the following policy after replacing the ``: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:Get*", "s3:List*", "s3-object-lambda:Get*", "s3-object-lambda:List*" ], "Resource": [ "arn:aws:s3:::*" ], } ] } ``` Ideally that bucket should only have event data that growthbook needs to calculated its metrics and no other data. You can further restrict the resources as your security policy requires. Afterwards your IAM user's permission page might look like: ![Athena permissions](/images/guides/athena-permissions.png) ### Self hosted We recommend creating a new IAM role with the same permissions as for Growthbook Cloud. This role can then be [attached to the ec2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#attach-iam-role) that is running Growthbook. ## Providing Credentials to Growthbook ### Growthbook Cloud If you are using Growthbook Cloud you will need to create an IAM user that has the above policies attached - either directly or through a group. You must then create an access key by clicking on Security Credentials then "Create Access Key". You can then choose "Third-party service". It will warn you that this is not best practice, but unfortunately this is the only way to give Growthbook access at the moment. We are working on other ways to connect in the future. You can then confirm and click next. You can add a tag if you like and then press "Create access key". You should see then see following screen: ![Athena access keys](/images/guides/athena-keys1.png) In another browser tab open up Growthbook Data Sources tab and choose your event tracker. Select Athena as your data source type. You can then copy the Access Key and Secret Access Key from the AWS browser tab to their corresponding fields. ### Self hosting If you are self hosting then in addition to the method above you can also pass the credentials in via environmental variables or part of the instance metadata. You can select which method you want in the `Authentication Method` field. ## Remaining Configuration `AWS Region` - This should be the AWS Region your Athena database is in. From the AWS console you would see it on the right side of the search bar on the top of the screen next to the account name. `Workgroup` - This is the workgroup within Athena. `Default Catalog` - This is that catalog where the event data lives. `Default Database` - This is the database where the event data lives. ![Extra Fields in Growthbook for setting up Athena](/images/guides/athena-extra-fields.png) `S3 Results URL` - This is the s3 URL where the results to Athena queries get saved. When setting up Athena for the S3 results url, we recommend naming your bucket with the prefix `aws-athena-query-results-` as the AWSQuicksightAthenaAccess gives permission to write to any bucket with this prefix. If Growthbook warns you that it can not write to an s3 location other than the one you select here it is most likely because you have set the [workgroup to override client side settings](https://docs.aws.amazon.com/athena/latest/ug/workgroups-settings-override.html). If that is the case you would either need to change that setting or add the permissions for growthbook to also write to the s3 results url saved there. --- ## GrowthBook and BigQuery import MaxWidthImage from "@site/src/components/MaxWidthImage"; import CommercialFeature from '@site/src/components/CommercialFeature'; This document outlines the steps needed to add your BigQuery database to GrowthBook. ## 1. Create a service account for GrowthBook Within your [Google Cloud console account](https://console.cloud.google.com/iam-admin/serviceaccounts), create a service account for GrowthBook to use ![Create service account button in BigQuery](/images/guides/bigquery-1-addserviceaccount-for-gb-highlited.png) ![Create a new service account in BigQuery](/images/guides/bigquery-2-addserviceaccount-for-gb3.png) Create a service account name and account ID. On the next page you need to add 3 specific roles: ![Grant roles for service account in BigQuery](/images/guides/bigquery-3-addserviceaccount-for-gb4-roles.png) On the Grant page, add the following three permissions roles for read-only access: - BigQuery Data Viewer - BigQuery Metadata Viewer - BigQuery Job User ![Additional roles granted for service account](/images/guides/bigquery-4-addserviceaccount-for-gb5-roles.png) On the final page when creating a service account, you can skip the optional fields. You should see the new service account listed, without a `Key ID`. We need to add an access key to this account so the credentials can be added to GrowthBook. Click on actions, and select `Manage Keys`. ![New account listed](/images/guides/bigquery-5-getjson-key.png) There are two ways to provide credentials to GrowthBook: - Auto-discovery from environment variables or GCP metadata (only available when self-hosting) - Upload a JSON key file for the service account We're going to show how to do the JSON key file method. On the keys page, add a new key, and select JSON. ![Create new key from dashboard](/images/guides/bigquery-6-getjson-key2.png) ![Create private key for GrowthBook popup](/images/guides/bigquery-6-getjson-key3.png) This will cause the JSON key to be downloaded to your computer. ## 2. Connect GrowthBook to BigQuery From the Metrics and Data → Data Source page, click on add new data source and select the event tracker you're using. If your event tracker is not listed, or you're using something custom, click on the "Custom" button at the bottom. Selecting an event tracker here will pre-populate the experiment exposure query which is need to determine which user saw which experiment variation. Depending on your needs, you may still need to adjust these queries to match your specific schema. ![Add BigQuery to GrowthBook](/images/guides/bigquery-7-add-datasource1.png) Select BigQuery as the data source type. ![Selecting BigQuery as the data source type.](/images/guides/bigquery-7-add-datasource2.png) Add the names you'd like to use, and select the JSON key file that was downloaded earlier. Growthbook will use the `Project Id` and `Dataset` you enter as the default ones when creating queries. You can get the value for these fields from the [Google Cloud explorer](https://console.cloud.google.com/bigquery). You will see the top level project id, and when expanded, find the dataset which has your experiment exposure table (which will be `experiment_viewed` if you use Segment or Rudderstack). ![Get default project id and default dataset](/images/guides/bigquery-8-getdefault-names.png) When you click save, GrowthBook will test the connection to make sure the credentials are correct. If the connection is successful, you should see a success message on the next page. ## Enabling Data Pipeline Mode Pipeline mode can reduce query costs if you grant the GrowthBook service account write permissions in your data warehouse. [More details can be found here.](/app/data-pipeline) ## Monitoring GrowthBook query cost Whenever we query your BigQuery database we add `{ integration: "growthbook" }` as a label to the query job to make it easy for you to monitor cost or filter GrowthBook query jobs by label for other use cases. Read more about how to [group by label value for a specific key here.](https://cloud.google.com/billing/docs/how-to/bq-examples#group_by_label_value_for_a_specific_key) ## Troubleshooting For help debugging missing experiment data in your BigQuery instance, please see our [Google BigQuery (BQ) Troubleshooting Guide](/kb/google-analytics/google-big-query-bq-troubleshooting#initial-sync-delay-for-new-ga4-properties) and our our [Google Analytics (GA4) Troubleshooting Guide](/kb/google-analytics/google-analytics-ga4-troubleshooting) --- ## Setting up Clickhouse as a data source import MaxWidthImage from "@site/src/components/MaxWidthImage"; Connecting to Clickhouse is very straightforward. You just need to provide the connection string and credentials. If you are using something like Clickhouse Cloud, you can find the connection string in the UI. ![Clickhouse Cloud Connection String](/images/guides/guide-clickhouse-1.png) If you are making connection credentials just for GrowthBook, you can give it read-only access to the database. If you are using GrowthBook Cloud (app.growthbook.io), make sure to whitelist the ip address 52.70.79.40 if applicable. --- ## Setting up Databricks as a data source import MaxWidthImage from "@site/src/components/MaxWidthImage"; To connect to Databricks, you need to provide GrowthBook with the connection details for your SQL Warehouse along with credentials that let GrowthBook authenticate and run queries against it. ## 1. Find the connection details for your SQL Warehouse In your Databricks instance, navigate to the SQL Warehouses, select your SQL Warehouse that stores the data you want GrowthBook to access, and the click Connection Details. You should see the following page, which contains most of the fields needed to connect GrowthBook to your Databricks SQL Warehouse. ![Finding SQL Warehouse Connection Details in Databricks](/images/guides/databricks-connection.png) From this page you will use the **Server Hostname**, **Port**, and **HTTP Path** values when adding the data source to GrowthBook. ## 2. Choose an authentication method GrowthBook supports two ways to authenticate against Databricks: - **OAuth (machine-to-machine / service principal)** — the recommended method. GrowthBook uses a Databricks service principal's Client ID and OAuth secret to obtain short-lived access tokens automatically at query time. - **Personal Access Token (PAT)** — the legacy method. Databricks now recommends OAuth over personal access tokens because OAuth provides stronger security. Whichever method you choose, make sure the service principal or user has permission to execute SQL queries against your SQL Warehouse. ### Option A: OAuth (machine-to-machine) OAuth M2M authenticates using a Databricks service principal. GrowthBook exchanges the service principal's Client ID and OAuth secret for short-lived tokens (via the `client_credentials` grant) and refreshes them automatically, so you never store a long-lived token. 1. Create a Databricks service principal (or reuse an existing one). See the Databricks docs on [managing service principals](https://docs.databricks.com/en/admin/users-groups/service-principals.html). 2. Generate an OAuth secret for that service principal. Databricks shows the **Client ID** (the service principal's application ID) and lets you generate a **Secret**. Copy both now — the secret is only shown once. See [OAuth machine-to-machine (M2M) authentication](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m). 3. Grant the service principal permission to use your SQL Warehouse and access the data you want to query. 4. In GrowthBook, you will enter the **Client ID** and **OAuth Secret** in step 3 below. ### Option B: Personal Access Token There are two kinds of access tokens that will work to connect GrowthBook to Databricks: - A personal access token for a user account - A personal access token for a service principal Follow these instructions to create a service principal and create an access token for the service principal: https://docs.databricks.com/en/dev-tools/service-principals.html Alternatively, you can create a personal access token for a Databricks workspace user following these instructions: https://docs.databricks.com/en/dev-tools/auth.html#pat ## 3. Add credentials to GrowthBook Now that you have the connection details and credentials, you can add the data source in GrowthBook. Click "Add a Datasource" from the Data Sources page under Metrics and Data and either select your event tracker or click "custom data source". Then you can select Databricks as your data warehouse and enter the above credentials. Note: We only use the event tracker (e.g. Snowplow, Segment) to help us build queries for you. This information **does not** impact how or whether we are able to connect to Databricks. --- ## Missing Datasource Docs import Link from '@docusaurus/Link'; export default function MissingDatasourceDocs({ datasource, slug }) { return ( When setting up {datasource} to work with Growthbook, remember it's important to create read-only users with minimal permissions, ideally granting only read access to the relevant data tables that need to be aggregated. If you are using GrowthBook Cloud (app.growthbook.io), make sure to whitelist the ip address 52.70.79.40 if applicable. Unfortunately there isn't a specific guide for setting up {datasource} for Growthbook yet. However, if you encounter any difficulties while setting up your data source, you can seek assistance from our active developer community on Slack. They can help you troubleshoot and ensure your implementation is successful. Once you've successfully set up {datasource} according to best practices, we encourage you to contributing to the community by creating improved documentation for {datasource}. Your contribution will benefit other users in the future. ); } --- ## Setting up Mixpanel as a data source :::caution Deprecation notice Using Mixpanel as a direct datasource is no longer supported. This is because because Mixpanel has placed JQL, their query language, in [maintenance mode](https://docs.mixpanel.com/docs/reports/apps/jql). In order to use Mixpanel with GrowthBook, we suggest you set up an export to a data warehouse following [Mixpanel's documentation](https://docs.mixpanel.com/docs/data-pipelines) and then use that separate data warehouse (e.g. Snowflake, BigQuery) as your Data Source in GrowthBook. ::: --- ## Setting up MS SQL or SQL Server as a data source import MissingDatasourceDocs from './missing-datasource-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## Setting up MySQL or MariaDB as a data source import MissingDatasourceDocs from './missing-datasource-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## Setting up Postgres as a data source import MissingDatasourceDocs from './missing-datasource-docs.mdx' export const { title, description, sidebar_label, slug } = frontMatter --- ## Setting up Prestodb or Trino as a data source import MaxWidthImage from "@site/src/components/MaxWidthImage"; When you create a connection from GrowthBook to Prestodb or Trino, you there are a few connection options. ![Connecting to Prestodb or Trino in GrowthBook](/images/integrations/presto/presto-trino-connection-settings.png) :::note GrowthBook uses the presto-client-node library to connect to Prestodb or Trino. Find more information about the library [here](https://www.npmjs.com/package/presto-client). ::: ## Engine This option lets you tell the client that your data source is `presto` or `trino`. For `trino` data sources, the connection library adds additional compatibility information to work correctly with `trino` data sources. The default is `presto`. ## Authentication Options ### Basic Auth If your Prestodb or Trino instance is configured with basic auth, you can provide the username and password in the connection settings. ### Custom Auth For custom authentication, GrowthBook will connect to your Prestodb or Trino instance using the provided HTTP Authorization header with the provided string. ### None If your Prestodb or Trino instance does not require authentication, or you are doing authentication based on IP or other methods, you can select "None" in the connection settings. This is not recommended for Cloud instances of GrowthBook. In these cases, the 'user' will be set to 'growthbook'. ## Host The url to access your Prestodb or Trino instance. This should include the protocol (http or https). ## Port Presto coordinator port. The default is 8080. ## Default Catalog and Schema GrowthBook will create default queries using the catalog you provide here. If you update this value later, you may have to update existing queries to use the new catalog. ## source The source field is used to help identify where the connection to presto is coming from. By default, this is set to 'GrowthBook'. --- ## Setting Up Amazon Redshift as a Data Source in GrowthBook import MaxWidthImage from "@site/src/components/MaxWidthImage"; Connecting Amazon Redshift to GrowthBook allows you to analyze experiment data and metrics directly from your data warehouse. :::note This guide focuses on Redshift Serverless, but the steps are similar for provisioned Redshift instances. ::: ## 1: Find Connection Details 1. In the **AWS Management Console**, navigate to **Amazon Redshift** → **Serverless**. 2. Click on **Workgroup configuration**, select your workgroup. 3. Under **General information**, locate the **Endpoint**, which looks like this: `default-workgroup.1234567890.us-east-1.redshift-serverless.amazonaws.com:5439/dev` This string comprises the following parts: - Host: `default-workgroup.1234567890.us-east-1.redshift-serverless.amazonaws.com` - Port: `5439` - Database Name: `dev` ![Redshift details](/images/guides/redshift-details.png) ## 2: Configure Security settings 1. Go to **AWS Console** → **Amazon Redshift** → **Serverless** → **Workgroups**. 2. Click on your Workgroup and scroll to **Network and Security**. 3. Click **Edit** and check **Turn on Publicly accessible**. ![Redshift Publicly Accessible](/images/guides/redshift-publicy-accessible.png) 4. Click **Save Changes**. 5. Click on the **VPC security group** link. 6. Select the Security Group matching your Workgroup. 7. Edit **Inbound Rules**: - Protocol: TCP - Port: 5439 - Source: 52.70.79.40/32 (GrowthBook’s IP) 8. Click **Save Changes**. ![Redshift Inbound Rules](/images/guides/redshift-inbound-rules.png) :::caution Enabling public access makes your database accessible from the internet. Ensure your security group settings are restrictive. ::: ## 3: Create a Read-Only User To prevent unauthorized changes, use a dedicated read-only user for GrowthBook. 1. Open AWS Query Editor or your preferred SQL client. 2. Run the following: ```sql CREATE USER growthbook_user WITH PASSWORD 'securepassword'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO growthbook_user; ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO growthbook_user; ``` :::tip If your data is in a different schema (e.g., analytics), replace public with your schema name. ::: 3. Use this user when connecting GrowthBook. ## 4: Add Redshift as a Data Source in GrowthBook 1. In GrowthBook, go to **Settings** → **Data Sources.** 2. **Click Add Data Source** and choose **Redshift**. 3. Enter the details from Step 1: - Host: (Your Workgroup Endpoint for Serverless or Cluster Endpoint for Provisioned) - Port: `5439` - Database: (Your database name, e.g., `dev`) - User: `growthbook_user` - Password: (Your secure password) - Require SSL: Recommended but may depend on your Redshift setup. 4. Click **Next** to test the connection. ![Redshift GrowthBook connection](/images/guides/redshift-connection-in-gb.png) 🚀 Now, GrowthBook can securely query your Redshift database! --- ## Setting up Snowflake as a data source import CommercialFeature from '@site/src/components/CommercialFeature'; We support multiple [account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) formats when connecting to Snowflake. An example account identifier is `xy12345.us-east-2.aws`. ## Self-Hosting If you are self-hosting GrowthBook, you can send queries to Snowflake through an Authenticated Proxy. To enable this, set a `SNOWFLAKE_PROXY` environment variable in your GrowthBook container. Here is an example: ``` SNOWFLAKE_PROXY=http://username:password@proxyserver.company.com:80 ``` ## Enabling Data Pipeline Mode [More details can be found here.](/app/data-pipeline) --- ## Setting up Vertica as a data source import MaxWidthImage from "@site/src/components/MaxWidthImage"; To connect GrowthBook to Vertica, GrowthBook must be able to connect to your Vertica instance and run queries on your behalf in your Vertica warehouse. This guide walks you through connecting GrowthBook to your Vertica instance. ## 1. (optional) Create a user in Vertica for GrowthBook queries Create a user in Vertica for the purpose of connecting GrowthBook to your Vertica warehouse. You can re-use an existing user, but using a dedicated user is recommended to keep access control and limited to necessary permissions. This user needs to: - have SELECT permissions in the schema that houses your experiment assignment and metric data. - be able to be authenticated via the HOST method from the cloud GrowthBook server (or your own GrowthBook instance if self-hosting). This can be achieved by granting the 'password' method authentication to the user. ## 2. Create a new Data Source in GrowthBook Navigate to _Metrics and Data → Data Sources_ to create a new Data Source. In the creation flow, you can either select your tracker of choice in the "Guided Setup" or select "Manual Set Up". The guided setup allows GrowthBook to use well known tracker schemas to pre-specify certain queries. If your tracker isn't available or you have a custom schema for your event data, pick a manual set up and you will be able to modify your queries later. In either case, select "Vertica" as the warehouse type. ## 3. Enter your Vertica connection details Once you select "Vertica" you will enter the credentials to connect GrowthBook to your warehouse. We need a host that GrowthBook can connect to, a port, and the user credentials from Step 1. You can additionally use SSL to validate your connection. We also require a "Database". This field helps us populate a schema browser that makes writing queries in GrowthBook easier. ![Vertica warehouse connection](/images/vertica.png) When you save your Datasource, we will immediately test the connection with a simple `"SELECT 1"` query to ensure the connection is working. ## 4. Begin setting up your experiment assignment queries and metrics You can now begin modifying or creating your [Experiment Assignment Queries](/app/datasources#experiment-assignment-queries) and [Fact Tables](/app/metrics) to define your Metrics. --- ## Webhooks Events import EventWebhookList from '@site/src/partials/event-webhook/_event-webhook-list.md' We currently support the following event types: --- ## Event Webhooks import ExternalLink from '@site/src/components/ExternalLink' import Pill from '@site/src/components/Pill'; # Event Webhooks beta GrowthBook has event-based webhooks that trigger a script on your server whenever something changes within GrowthBook. ## Adding a Webhook When logged into GrowthBook as an admin, navigate to **Settings → Webhooks**. There you can add a webhook endpoint and select which events you want to be notified about. ## Supported Event Types See the [full list of events](/app/webhooks/event-webhooks/events). ## HTTP parameters When creating or updating a webhook, you can select which HTTP method should be used. Available methods are: `POST`, `PUT` or `PATCH`. You can also add your own custom headers. The format of custom headers is a JSON object of the form: ``` { [Header-Name]: "Header Value", ... } ``` ## Slack notifications Webhooks can be used to send notifications to Slack channels and users with much less overhead than setting up a whole Slack integration! Follow our instructions on [setting up a Slack integration](/integrations/slack) to create a Slack application and connect it to GrowthBook events. ## Discord notifications Discord notifications work the same way as Slack notifications. The only difference is that you need to follow our instructions on [setting up a Discord integration](/integrations/discord). ## Projects, tags and environments filtering You can select which notifications should be sent to your webhook according to projects, tags or environments. When set, your webhook will only fire when events associated with the selected items are triggered. If unset, you webhook will fire on all events regardless of the given item. For instance, if your webhook is configured to fire on `feature.created` events with `Project Apollo` as a project filter, only `feature.created` events related to `Project Apollo` (and any other selected project) will trigger your webhook. If the projects filter is empty, your webhook will fire for all `feature.created` events on any project. ## Testing your webhook Once your webhook is configured, you should be able to test it! Look for the `Test` button on the webhook's settings page. When clicked, a test event should be delivered to the configured endpoint. This can be quite helpful to debug and confirm that all the webhook's parameters have been set correctly. ## Examples - [Web hooks implementation example ](https://github.com/growthbook/examples/tree/main/webhooks-impl) ## Errors and Retries If your endpoint returns any HTTP status besides `200`, the webhook will be considered failed. Failed webhooks are tried a total of 3 times using an exponential back-off between attempts. You can view the status of your webhooks in the GrowthBook app under **Settings → Webhooks**. ### VPCs and Firewalls If your webhook endpoint is behind a firewall and you are using GrowthBook Cloud, make sure to whitelist the ip address `52.70.79.40`. ## Verify Signatures Webhook payloads are signed with a shared secret so you can verify they actually came from GrowthBook. The signature is passed in a `X-GrowthBook-Signature` header. You can find the signature of a given webhook on its settings page. Look for a random string starting with `ewhk_...`. Here is example code in NodeJS for verifying the signature. Other languages should be similar: ```js const crypto = require("crypto"); const express = require("express"); const bodyParser = require("body-parser"); // Retrieve from GrowthBook settings const GROWTHBOOK_WEBHOOK_SECRET = "abc123"; const port = 1337; const app = express(); app.post( "/webhook", bodyParser.raw({ type: "application/json" }), (req, res) => { const payload = req.body; const sig = req.get("X-GrowthBook-Signature"); const computed = crypto .createHmac("sha256", GROWTHBOOK_WEBHOOK_SECRET) .update(req.body) .digest("hex"); if (!crypto.timingSafeEqual(Buffer.from(computed), Buffer.from(sig))) { throw new Error("Signatures do not match!"); } const data = JSON.parse(payload); // TODO: Do something with the webhook data // Make sure to respond with a 200 status code res.status(200).send(""); } ); app.listen(port, () => { console.log(`Webhook endpoint listening on port ${port}`); }); ``` ## SDK Webhooks (deprecated) GrowthBook has another type of webhook, meant specifically to keep SDKs up-to-date with the latest feature flag states. These have been deprecated in favor of SDK Connections and the GrowthBook Proxy Server. The payload for these legacy webhooks are described below for reference. ### Payload SDK Webhooks will do a `POST` to the endpoint you provide. The body is a JSON object containing feature definitions in the same format that SDKs are expecting. Here's an example payload: ```json { "timestamp": 1625098156, "features": { "feature1": { "defaultValue": true } } } ``` The `features` field has one entry per feature definition. Features can have the following properties: - **defaultValue** - **rules[]** - Array of feature rules - **condition** - A JSON condition using MongoDB query syntax - **force** - Force a specific value, takes precedence over all other rules besides `condition` - **variations[]** - Run an experiment and randomly assign one of the specified variations - **key** - When running an experiment, this is the experiment key that will be passed to the tracking callback function - **weights[]** - Determines how traffic is split between variations in an experiment - **coverage** - Specifies what sampling rate (0 to 1) to use for including users in an experiment. A rate of `1` means everyone is included. A rate of `0` means no one is. --- ## Global SDK Webhooks :::info Global SDK Webhooks are only available for self-hosted GrowthBook installations. ::: Global SDK Webhooks are just like [SDK Webhooks](/app/webhooks/sdk-webhooks), but configured via environment variables instead of the GrowthBook UI. When any SDK Connection payloads changes in any organization, all of your Global SDK Webhooks will be triggered. ## Setup Define a `WEBHOOKS` environment variable as a JSON string of an array of global webhook objects. The only required field for a global webhook is **`url`**. Here's a minimal example: ``` [{"url":"https://example.com"}] ``` There are additional fields you can specify: - **`signingKey`** (string) - Will be used to add a signature header that enables you to verify the webhook origin - **`method`** (string) - One of `GET`, `PUT`, `POST`, `PURGE`, `PATCH`, or `DELETE`. Defaults to `POST` if omitted - **`headers`** (object) - Additional headers to add to the webhook request. Useful for adding auth headers for example. Default `{}`. - **`payloadFormat`** (string) - How to format the body (ignored when method = "GET"). One of `standard`, `standard-no-payload`, `sdkPayload`, `edgeConfig` or `none`. Defaults to `standard`. Read more about the different formats in the SDK Webhooks [Payload Format docs](/app/webhooks/sdk-webhooks#payload-format). - **`payloadKey`** (string) - Only used when payloadFormat is set to `edgeConfig`. The key in Vercel Edge Config you want to update. _Deprecated:_ - **`sendPayload`** (boolean) - Whether or not to include the full SDK Payload in the body. `true` maps to `payloadFormat = standard`; false maps to `payloadFormat = standard-no-payload`. Here's a full example using all of the fields: ``` [{"url":"https://example.com","signingKey":"abc123","method":"PUT","headers":{"X-Custom-Header":"foo"},"payloadFormat":"sdkPayload"}] ``` ## Verify Signatures If you define a `signingKey`, you can use it to verify that webhooks are coming from GrowthBook. **See: [SDK Webhooks - Verify Signatures](/app/webhooks/sdk-webhooks#verify-signatures)** ## Errors and Retries Unlike normal SDK Webhooks, Global SDK Webhooks are NOT retried on error. On failure, they will output container logs as well as save them to the Mongo `sdkwebhooklogs` collection. --- ## Webhooks Overview GrowthBook provides a variety of different outbound webhooks, all of which enable push communication from the GrowthBook server to applications. We will briefly discuss the different types of webhooks that GrowthBook supports and when you might want to use them. ### Event Webhooks **See: [Event Webhooks](/app/webhooks/event-webhooks)** Access via: **Settings → Webhooks** These trigger when GrowthBook's state changes. For example, sending a detailed message about how a feature was modified or a new experiment was created. Filterable by project, environment, and event type(s). ##### Common Use Cases: - Pinging an internal monitoring service - Maintaining a custom audit log - Messaging on Slack or Discord Event webhooks can be formatted for Slack and Discord out of the box. For integration details, see: - [Slack Integration](/integrations/slack) - [Discord Integration](/integrations/discord) ### SDK Webhooks **See: [SDK Webhooks](/app/webhooks/sdk-webhooks)** Access via: **SDK Configuration → SDK Connections**, choose a connection, click **Add Webhook** These are tightly coupled with your SDK Connections. They trigger whenever the connection's SDK payload (feature and experiment definitions) changes. They can optionally send the new SDK payload. ##### Common Use Cases: - Updating or flushing CDN cache - Updating a cache microservice - Pushing feature/experiment updates to your application ### Global SDK Webhooks **See: [Global SDK Webhooks](/app/webhooks/global-sdk-webhooks)** Global SDK Webhooks are limited to self-hosted users only. These are similar to SDK webhooks, but are not limited to a single SDK Connection. Instead, GrowthBook will fire _all_ SDK Connection changes to one or more webhooks, which are defined via environment variables. ##### Use Cases: - For larger organizations, this saves the trouble of needing to manually configure hundreds of individual SDK Webhooks for each connection. - This pattern is especially common for multi-org installations of GrowthBook. ### GrowthBook Proxy Webhook **See: [GrowthBook Proxy documentation](/self-host/proxy)** Similar to SDK webhooks, this webhook is a special use webhook for communicating with the GrowthBook Proxy. Unlike SDK webhooks, proxy webhooks are not customizable. - Self-hosted users should configure a single global proxy webhook using environment variables. - Cloud users may optionally configure a GrowthBook Proxy webhook for each SDK Connection. ## Webhook Secrets GrowthBook includes a secure way to include a secret token in your webhook requests, either in the endpoint or in a header. On the **Settings -> Webhooks** page, you can define webhook secrets with a key and value. You can then reference this secret by key within your webhook URL or in the headers using a simple Handlebars syntax (`{{ MY_SECRET_KEY }}`). For example, if you define a secret with the key `SECRET_TOKEN`, you can use it in your webhook URL like this: ``` https://api.example.com/webhook?secret={{SECRET_TOKEN}} ``` The secret value will be swapped in right before sending the request and will never be visible within the GrowthBook UI or logs. ### Restricted Origins For better security, you can restrict the origins where a webhook secret can be used. When enabled, the secret will only be available for webhooks where the destination URL matches one of the allowed origins. ## Notable Mentions Less common or soon-to-be-deprecated webhooks: ### Legacy Webhooks These function similarly to SDK Webhooks. They should not be used going forward; existing legacy webhooks will likely be migrated to SDK Webhooks during a future GrowthBook version release. ### Fastly CDN Purge Webhook If you are using Fastly as your CDN, and have defined surrogate keys for each SDK endpoint, you can pass `FASTLY_SERVICE_ID` and `FASTLY_API_TOKEN` into your environment variables to enable automatic cache purging. However, we recommend using SDK Webhooks for more control and flexibility. --- ## SDK Webhooks # SDK Webhooks GrowthBook has SDK-based webhooks that trigger a script on your server whenever something changes within GrowthBook which will affect that SDK. ## Adding a Webhook When logged into GrowthBook as an admin, navigate to **SDK Connections** and select an existing SDK Connection. Under the SDK Webhooks section you can add a webhook. There are built-in webhook types for syncing to **Cloudflare KV**, **Fastly KV**, and **Vercel Edge Config**. For everything else, you can configure a generic **HTTP Endpoint** to be hit. Once a SDK webhook is created you will be able to view the status and fire a test event. ### VPCs and Firewalls If your webhook endpoint is behind a firewall and you are using GrowthBook Cloud, make sure to whitelist the ip address `52.70.79.40`. ## Verify Signatures SDK Webhook payloads are signed with a shared secret so you can verify they actually came from GrowthBook. ### Standard Webhooks We follow the [Standard Webhooks](https://www.standardwebhooks.com/) specification, so you can use any of their SDKs to verify our webhook signatures. ```ts import { Webhook } from "standardwebhooks" const wh = new Webhook(base64_secret); wh.verify(webhook_body, webhook_headers); ``` ### Custom Verification Webhook requests sent to your endpoint include 3 headers: - `webhook-id` - The unique id for this event - `webhook-timestamp` - The unix integer timestamp of the event - `webhook-signature` - The signature (format described below) To create the signature, we concatenate the `webhook-id`, the `webhook-timestamp`, and the body contents, all separated by dots (`.`). Then, we create an HMAC SHA-256 hash of this using the shared secret. What we set in the `webhook-signature` header is the hashing algorithm identifier for HMAC SHA-256 (`v1`), followed by a comma (`,`), followed by the base64-encoded hash from above. For example: ``` v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4= ``` You can find the shared secret via **SDK Configuration → SDK Connections**, choosing the connection, and viewing your webhook's details. Here is example code in NodeJS for verifying the signature. Other languages should be similar: ```js const crypto = require("crypto"); const express = require("express"); const bodyParser = require("body-parser"); // Retrieve from GrowthBook SDK connection settings const GROWTHBOOK_WEBHOOK_SECRET = "wk_123A5341464B3A13"; const port = 1337; const app = express(); app.post( "/webhook", bodyParser.raw({ type: "application/json" }), (req, res) => { // If there is no body sent, use an empty string to compute the signature const body = req.body || ""; // Get the request headers const id = req.get("webhook-id"); const timestamp = req.get("webhook-timestamp"); const rawSignature = req.get("webhook-signature") || ""; // Remove the "v1," prefix from the signature for comparison const signature = rawSignature.split(",")[1]; if (id && timestamp && signature) { // Compute the signature const computed = crypto .createHmac("sha256", GROWTHBOOK_WEBHOOK_SECRET) .update(`${id}.${timestamp}.${body}`) .digest("base64"); if (!crypto.timingSafeEqual(Buffer.from(computed), Buffer.from(signature))) { throw new Error("Invalid signature"); } } else { throw new Error("Missing signature headers"); } const parsedBody = JSON.parse(body); const payload = parsedBody.data.payload; // TODO: Do something with the webhook data // Make sure to respond with a 200 status code res.status(200).send(""); } ); app.listen(port, () => { console.log(`Webhook endpoint listening on port ${port}`); }); ``` ## Errors and Retries If your endpoint returns any HTTP status besides `200`, the webhook will be considered failed. Webhooks are retried up to 2 additional times with an exponential back-off between attempts. You can view the status of your webhooks in the GrowthBook app under **SDK Connections**. ## Supported HTTP Methods - **GET** - **POST** - **PUT** - **DELETE** - **PURGE** - **PATCH** ## Payload Format For all methods other than `GET`, you may send a payload body. By default, webhooks will send in the "Standard" format. ### Standard Follows the **Standard Webhooks** specification. Inludes a JSON-encoded SDK Payload in the `data.payload` field. Example payload: ```json { "type": "payload.changed", "timestamp": "2024-04-03T01:54:20.449Z", "data": { "payload": "{\"features\":{\"my-feature\":{\"defaultValue\":true}}}" } } ``` The `data.payload` object contains the exact JSON format that our SDKs are expecting. For example, you can pass this directly into the JavaScript SDK: ```ts const payload = JSON.parse(parsedBody.data.payload); const gb = new GrowthBook(); await gb.init({ payload: payload }); ``` ### Standard (no SDK Payload) Same as above, but without the `data.payload` field. Example payload: ```json { "type": "payload.changed", "timestamp": "2024-04-03T01:54:20.449Z", } ``` ### SDK Payload Sends the raw SDK Payload using the same format as our SDK features endpoint. This is usually the correct format if you are using the webhook to set a cache value or assigning key/value storage. Example payload: ```json {"features":{"my-feature":{"defaultValue":true}}} ``` ### Vercel Edge Config Formats the body to work directly with Vercel's Edge Config API. Example payload: ```json { "items": [ { "operation": "upsert", "key": "gb_payload", "value": {"features":{"my-feature":{"defaultValue":true}}} } ] } ``` In order for this to work properly, you must specify the following options as well, replacing the `your_*` placeholders with actual values. - Endpoint URL = `https://api.vercel.com/v1/edge-config/your_edge_config_id_here/items` - HTTP Method = `PATCH` - Headers = `{ "Authorization": "Bearer your_vercel_api_token_here" }` - Payload Key = `your_value_key_here` (defaults to `gb_payload` if left blank) :::tip Updating Edge Config items owned by a team If you are updating an Edge Config item owned by a team, add `?teamId=your_team_id_here` to the end of your Endpoint URL. :::