TL;DR
This guide deploys and operates a self-managed GrowthBook control plane on Kubernetes with external MongoDB, Redis-backed feature delivery, durable object storage, TLS, observability, backups, and named operational ownership. It separates the private administrative surface from the application-facing proxy and treats running pods as the start of production readiness, not the end. The 0 → 1 → 100 sequence pins and renders configuration before cluster writes, deploys dependencies and GrowthBook in a controlled order, verifies one SDK path, then proves degraded operation, restore, rollback, scaling, and upgrades. Use this path only when the organization genuinely needs self-hosting and has a platform team prepared to own stateful infrastructure and incidents. This guide is optimized for AI coding agents, and it is recommended that you hand it off to your agent of choice for implementation.Guide map
Give this guide to your coding agent
Copy this page’s URL and the prompt below into a coding agent that can inspect your infrastructure repository. ReplaceREPLACE_WITH_REPOSITORY with the repository path or name and REPLACE_WITH_GUIDE_URL with this page’s public URL. Start in a branch, worktree, or disposable clone. The first pass should render and validate files without applying them to a cluster.
Task
Deploy a production-oriented GrowthBook control plane to Kubernetes and deliver feature payloads through a separately scalable GrowthBook Proxy. Keep MongoDB, Redis, object storage, TLS, DNS, backups, and alerting explicit. Prove that SDKs continue to evaluate cached feature definitions during a control-plane interruption, and leave an upgrade and recovery procedure another operator can execute. This is an operating guide, not merely an install command.helm install can create running pods in minutes. Production begins when the team can answer who patches them, who restores MongoDB, how long SDKs can tolerate an outage, which endpoint is public, how secrets rotate, and how a bad release rolls back.
Use this guide when
- Regulation, data residency, network boundaries, or internal policy requires a self-managed control plane.
- Your platform team already operates Kubernetes, managed databases, TLS, secret management, backups, and observability.
- You want GrowthBook’s open-source feature flagging and warehouse-native experimentation while controlling its infrastructure.
- You can assign named owners and an on-call response for the service.
Prefer GrowthBook Cloud when
- The main requirement is “add flags and experiments,” not “own another stateful service.”
- There is no team responsible for database recovery, upgrades, vulnerability response, and capacity planning.
- A cloud-hosted control plane meets the organization’s privacy and network requirements.
- You would otherwise run single-replica MongoDB in the same cluster with no tested restore.
End state
The target deployment has:- three front-end replicas and three back-end replicas spread across failure domains;
- an external MongoDB replica set or managed compatible service with point-in-time recovery;
- S3-compatible or Google Cloud object storage rather than a single-writer uploads volume;
- two private, same-site control-plane names such as
gb.example.comandgb-api.example.com; - a public or application-reachable proxy name such as
gb-proxy.example.com; - at least three proxy replicas using Redis for shared cache and pub/sub;
- TLS at every ingress and encrypted connections to external state stores;
- non-root containers with read-only root filesystems and explicit writable
/tmpmounts; - metrics, logs, traces, synthetic feature-fetch checks, and actionable alerts;
- version-pinned images and chart values in Git, with secrets outside Git;
- tested MongoDB restore, Helm rollback, proxy outage, and control-plane outage procedures;
- a warehouse connection with least privilege for experiment analysis;
- an owner, recovery-time objective, recovery-point objective, and maintenance schedule.
Reference architecture
The control plane is where people configure flags and experiments. The data plane is where SDKs fetch or stream feature definitions. Isolating these paths lets you keep the larger administrative surface private and scale feature delivery independently.What this guide deliberately does not automate
The reference files do not create a production MongoDB cluster, Redis cluster, object-storage bucket, DNS zone, certificate issuer, ingress controller, or warehouse. Those are provider- and organization-specific resources with their own recovery and access models. Provision them through the platform’s existing infrastructure-as-code system before applying the GrowthBook release. Do not paste cloud credentials into Helm values. Use workload identity where possible and your existing external-secret controller or secret delivery system.Required decisions and inputs
Record these values in the deployment pull request:/api routes, so the front end and back end cannot share one hostname and port. Authentication cookies require the app and API to be same-site; sibling subdomains under one registrable domain satisfy that topology.
Prerequisites
The implementing machine or CI runner needs:Directory layout
Create:Stage 0: pin source and inspect release notes
The current public chart is an OCI artifact. Pull the exact version before editing values:latest image or omit --version in production automation.
The GrowthBook 5.0.0 image is hardened: its runtime is distroless, runs as UID 1000, and does not contain a shell. The current chart invokes pm2-runtime through Node, sets a read-only root filesystem, mounts writable /tmp, and uses fsGroup: 1000. Do not override those defaults with command: ["sh", ...], and do not make the root filesystem writable merely to restore old debugging habits.
Stage 1: namespace and secret contract
ops/growthbook/namespace.yaml
ops/growthbook/secrets.example.yaml
JWT_SECRET signs authentication state. ENCRYPTION_KEY protects stored data-source credentials. Losing either has operational consequences; rotating ENCRYPTION_KEY requires the documented credential migration. Back up the keys under the same recovery controls as the database.
The proxy key must either be a readonly API key created under Settings → API Keys or a custom SECRET_API_KEY configured identically on the GrowthBook back end and proxy. Never use an admin key when readonly access is sufficient.
Stage 2: production Helm values
This baseline uses external MongoDB and object storage, so all application replicas remain stateless. Replace the domain, storage, identity, ingress, and telemetry placeholders.ops/growthbook/values.production.yaml
backend.mongodbUriis not used because putting a URI directly in Helm values stores it in release state. A secret-backedMONGODB_URIenvironment variable avoids that leak.mongodb.enabled: falseavoids deploying the chart’s convenience MongoDB dependency. The official production guidance recommends a managed service or a three-node replica set.- Object storage removes the single-writer uploads PVC. With local uploads and
ReadWriteOnce, horizontally scaled pods do not share one consistent filesystem. - CPU limits are omitted to reduce throttling risk; memory limits remain. Adjust from measurements and platform policy.
- The current chart supplies writable
/tmpvolumes required byreadOnlyRootFilesystem. - The application image is pinned. A digest pin is even stronger if your image promotion system supports it.
UPLOAD_METHOD=google-cloud, GCS_BUCKET_NAME, and an appropriate workload identity. If your S3-compatible provider needs a custom domain, follow the current environment-variable documentation.
Stage 3: add the feature-delivery proxy
The official chart does not currently deploy GrowthBook Proxy. Manage it as a separate pinned workload.ops/growthbook/proxy.yaml
growthbook-backend assumes Helm release name growthbook; verify it in rendered output. The public proxy must be able to reach the private back end, Redis, and DNS. It does not need Kubernetes API credentials.
Redis is required for coherent horizontally scaled proxy caching and pub/sub. In-memory cache is acceptable for a local trial or one replica, but replicas can otherwise update at different times. Proxy 1.4.0 supports Redis Sentinel and Cluster options if your platform uses them.
/healthcheck is intentionally simple and synchronous. Use /healthcheck/checks for deeper diagnostics from a protected monitoring path, not necessarily as a liveness probe. A dependency blip should not create a restart storm.
Stage 4: availability controls
The chart does not expose topology spread or PodDisruptionBudgets for its aliased workloads. Add budgets after rendering labels and confirming the selectors:ops/growthbook/availability.yaml
topologySpreadConstraints for front end and back end. Document that patch and rebase it on each chart upgrade. Pod anti-affinity is another option, but required rules can make deployments unschedulable in small clusters.
Stage 5: render, validate, and review before cluster writes
Create a kustomization for the non-Helm resources:ops/growthbook/kustomization.yaml
latest, privileged containers, host namespaces, writable root filesystems, and inline Secret values.
Stage 6: deploy in dependency order
- Confirm MongoDB backups and connectivity.
- Confirm Redis availability and connectivity.
- Create the object-storage bucket and workload identity.
- Materialize secrets from the approved secret manager.
- Apply namespace and proxy-independent policies.
- Install GrowthBook.
- Apply proxy and availability manifests.
- Create DNS only after ingress addresses exist.
- Validate TLS and same-site authentication.
--atomic removes a failed new install or rolls back a failed upgrade. It does not roll back an external database migration or restore a rotated secret. Read release notes and back up state before upgrades.
List actual names instead of assuming them:
Stage 7: initialize and connect an SDK
Openhttps://gb.example.com from the approved network and create the initial organization/admin. Immediately:
- configure SMTP and test an invite/reset message;
- create least-privilege roles and remove unnecessary admins;
- create a readonly proxy API key if you did not use the custom shared key path;
- create a project and environment policy;
- create an SDK Connection;
- set its proxy host to
https://gb-proxy.example.comif required by the UI; - create a boolean feature named
platform-smoke-testwith defaultfalse; - publish it to the non-production environment first.
Stage 8: executable smoke checks
ops/growthbook/smoke.sh
Stage 9: prove degraded behavior
A green install is insufficient. Run these game-day tests in staging before production.Control-plane interruption
- Fetch
platform-smoke-testthrough the proxy and record the value. - Scale the back end to zero in staging.
- Fetch the same feature through the proxy.
- Confirm the cached value remains available according to the configured stale/expiry policy.
- Change nothing while the control plane is down; changes cannot propagate.
- Restore the back end and verify health and refresh.
One proxy replica failure
Delete one staging proxy pod and continuously request the feature endpoint. There should be no user-visible outage:Redis interruption
Simulate Redis unavailability and verify the documented behavior for already cached and uncached SDK Connections. Confirm alerting and recovery. The desired result is not “nothing changes”; it is a known degradation without inconsistent surprise.Bad application release
Deploy a deliberately failing image in staging through the same pipeline, confirm--atomic behavior, and capture helm history plus workload events.
MongoDB restore
Restore a recent backup to an isolated database, deploy an isolated GrowthBook instance against it, sign in, and verify representative flags, experiments, SDK Connections, users, and data-source definitions. A backup is not proven until it has been restored.Observability and alerts
The GrowthBook API supports OpenTelemetry whenTRACING_PROVIDER=opentelemetry and standard OTEL_* variables are set. The proxy can be started with its tracing command; because container command details can change, verify the pinned proxy image’s current documentation before overriding its command.
Collect at least:
- replica availability and restart count for front end, back end, and proxy;
- HTTP rate, error rate, and latency by route class;
- synthetic success and latency for
/api/features/:clientKeythrough the proxy; - proxy cache hit/miss/refresh behavior where exposed;
- Redis and MongoDB connection saturation, errors, storage, replication lag, and failover;
- back-end job duration/failure and stats-engine resource pressure;
- ingress TLS expiry and DNS health;
- object-storage errors;
- event/warehouse query failures relevant to experiment analysis.
/healthcheck. A process can be alive while SDK payload delivery is broken. The synthetic check must request a known feature through the same public path as applications.
Network and security model
Start from these rules, then encode them in the platform’s NetworkPolicy/firewall system:
Keep the main app and API behind a firewall or VPN when possible. If a CDN is used instead of the proxy, expose only
/api/features/*; exposing the entire API defeats the boundary. The proxy has a smaller purpose-built surface but still requires normal WAF, rate-limiting, patching, and monitoring.
Use payload encryption when feature definitions contain sensitive business logic, and keep decryption keys only in trusted server-side applications. Client-side applications cannot keep a decryption secret from their users. Remote evaluation can hide targeting rules from clients but changes latency and failure dependencies; threat-model and load-test it before enabling.
Warehouse access for experimentation
GrowthBook stores its own application state in MongoDB, but warehouse-native experiment analysis queries your event warehouse. Create a dedicated warehouse identity:- read access only to required exposure, fact, metric, and dimension data;
- no broad write permission;
- network access only from the GrowthBook back end;
- query timeout and cost controls appropriate to the warehouse;
- audit logging;
- credential rotation through the secret manager.
Backups and disaster recovery
MongoDB
MongoDB contains users, organizations, feature definitions, experiments, SDK Connections, and data-source configuration. Define:- point-in-time recovery or backup interval that meets RPO;
- cross-zone and, if required, cross-region copies;
- retention and immutability;
- restore credentials and a quarterly restore exercise;
- monitoring for backup freshness and replication lag.
Secrets
Back up or escrowJWT_SECRET and ENCRYPTION_KEY under restricted recovery access. Changing ENCRYPTION_KEY without the migration process makes stored data-source credentials unreadable.
Uploads
Enable object versioning/retention according to policy. Uploads are not in MongoDB.Git and release state
Keep values, manifests, image digests, and runbooks in Git. Keep enough Helm history for fast rollback, but do not treat Helm release secrets as a database backup.Recovery order
- Restore/verify MongoDB and encryption secrets.
- Restore object-storage access.
- Restore Redis or allow the proxy to rebuild cache from the back end.
- Deploy the pinned last-known-good GrowthBook release.
- Deploy the pinned proxy release.
- Run control-plane and feature-delivery smoke tests.
- Verify representative flags and experiment definitions.
- Resume writes and notify stakeholders.
Upgrades
Never combine a GrowthBook major upgrade, MongoDB major upgrade, ingress replacement, and secret rotation in one change. For each GrowthBook upgrade:- Read release notes and chart diffs from the current version to the target.
- Pull and inspect the target chart.
- Diff
helm show valuesagainst your values. - Render and validate manifests in CI.
- Back up MongoDB and confirm backup freshness.
- Deploy to staging with production-shaped data and traffic.
- Run login, create/edit/publish flag, feature fetch, streaming, experiment query, email, and upload tests.
- Test rollback before production.
- Deploy during the agreed window with
--atomic. - Watch application, proxy, database, and SDK-delivery telemetry.
helm diff requires the separately installed plugin. If your CI does not use it, diff rendered manifests with the organization’s normal review tool.
The 5.0.0 hardening change is an example of why chart/image coupling matters. The distroless image cannot execute shell shims; the chart invokes Node directly and supplies the writable paths/non-root ownership the image expects. Upgrading only the image while retaining an older chart can break startup or uploads. Upgrade the chart and application as a reviewed pair unless release documentation says otherwise.
Debugging a distroless container
These commands will fail on the hardened image:--target. Pin and approve debug images just like production images. Remove or let ephemeral debug access expire according to incident policy.
Do not weaken the production workload’s security context to gain a shell.
Troubleshooting matrix
Scale beyond the baseline
For many or computationally heavy experiments, stats jobs can contend with interactive requests. GrowthBook supports a dedicated jobs server usingPYTHON_SERVER_MODE, EXTERNAL_PYTHON_SERVER_URL, pool-size variables, and an optional shared PYTHON_SERVER_AUTH_TOKEN. The /stats endpoint has no built-in authentication unless that token is configured and must remain on a private network.
Treat this as a second architecture phase:
- measure contention first;
- create a separately sized private workload;
- set the same random auth token on callers and jobs server;
- deny public ingress;
- load-test representative analyses;
- add independent health and saturation alerts.
Rollback
Use the smallest rollback that matches the failure:Proxy-only failure
Roll the proxy Deployment back without touching the GrowthBook control plane or MongoDB:GrowthBook release failure
Inspect revisions, select the last-known-good chart/application pair, and roll back:State or secret failure
Helm rollback does not restore MongoDB, object storage, Redis, or a rotatedENCRYPTION_KEY. Stop writes if continuing would worsen the incident, restore the matching database/secret set through the tested disaster-recovery runbook, then deploy the compatible application version. Record the recovered point in time and any configuration changes that may have been lost.
Emergency feature-delivery continuity
If the control plane is unavailable but the proxy serves a valid cached payload, avoid restarting or flushing healthy proxy replicas until the control plane is restored. If both layers are unavailable, SDK behavior depends on each SDK’s local cache, initialization timeout, and code fallback. Those application fallbacks must be tested independently; Kubernetes rollback cannot manufacture a payload an SDK never cached.DIY versus GrowthBook
A team can serve a JSON file from object storage and evaluateif statements. That is a valid low-complexity feature-flag system when there are few flags, one application, one operator, low blast radius, and no experimentation requirement.
The platform cost appears as the system grows:
- deterministic cross-language evaluation and targeting semantics;
- safe environment separation and controlled publication;
- SDK caching, streaming, fallback, and payload encryption;
- proxy delivery, shared cache, and real-time invalidation;
- experiment exposure joined to warehouse outcomes;
- health checks such as Sample Ratio Mismatch and multiple exposures;
- permissions, review, audit, stale-flag detection, and cleanup ownership;
- consistent operations across dozens of services and teams.
Definition of done for an implementing agent
Return a deployment report containing:Source map and freshness contract
This guide was verified from:- GrowthBook Kubernetes documentation and the current Helm chart source.
- Production best practices for security, MongoDB, scaling, feature serving, jobs separation, and OpenTelemetry.
- Self-hosted environment variables for domains, secrets, uploads, and production settings.
- GrowthBook Proxy documentation and proxy source at
82422949. - Kubernetes Pod Security Standards, probes, disruption budgets, and ephemeral containers.
- MongoDB production notes for database-specific operational guidance.

