Skip to content

feat(flagsmith): deploy, seed, and validate self-hosted Flagsmith - #20

Merged
amccabe merged 3 commits into
mainfrom
feat/flagsmith
Sep 23, 2026
Merged

amccabe merged 3 commits into
mainfrom
feat/flagsmith

Conversation

@amccabe

@amccabe amccabe commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

What changes

This PR adds self-hosted Flagsmith to example-stack: deployed by Flux from k8s/infra/flagsmith, seeded by setup.sh, checked by validate-stack.sh, and documented. It is inert at baseline: no flags are defined, and no service reads one.

  • Manifests. A flagsmith namespace, a one-replica Deployment of the unified image (docker.flagsmith.com/flagsmith/flagsmith:2.273.0, pinned by digest; the host redirects to Docker Hub, whose digest it is), a Service on ports 8000 and 9100, an Ingress on flagsmith.localhost, a demo-only Secret, a migrate-flagsmith-v1 Job, and a ServiceMonitor for the metrics server Flagsmith runs on 9100. The container runs serve, so migrations run only in the Job; readiness is /health/readiness/, which fails until the Job has applied the schema, and a startup probe covers the wait. One Gunicorn worker, 300Mi/300m requests, a 500Mi memory limit, and no CPU limit, like every other Deployment here: the chart's 500m throttled startup and fired CPUThrottlingHigh. The flagsmith database joins the PostgreSQL initdb ConfigMap, and the Flux Kustomization depends on apps.
  • Outbound defaults. ENABLE_TELEMETRY posts object counts to api.flagsmith.com on migrate; it is off. The dashboard reads its own feature flags from a Flagsmith, the vendor's hosted one by default, with a key the self-hosted bundle hardcodes; there is no off switch, and a failed fetch renders the maintenance page. Per Flagsmith's "Running Flagsmith on Flagsmith" docs, the seed creates a second organization and project, flagsmith-dashboard (the free plan allows one project per organization, self-hosted too), writes its client-side key to a flagsmith-dashboard Secret in the flagsmith namespace, and restarts the Deployment, which reads the key through an optional secretKeyRef and points the URL at the stack's own ingress. The dashboard then fetches its flags on loopback; the project needs no flags, since the frontend has built-in defaults. Until the seed's restart, a dashboard opened during bring-up uses the vendor default. The readme says so beside Grafana's and Forgejo's entries.
  • Seed. A bootstrap superuser through the signup endpoint (the image's bootstrap command gives no usable password), an organization and project named example-stack, and exactly the staging and production environments. --flagsmith-password sets the password, default example-stack-demo; later runs log in with it rather than resetting it. Request bodies are built in variables because bash 3.2 brace-expands JSON nested inside "$(...)".
  • Bootstrap Secrets. flagsmith-bootstrap holds the admin token and the two environment keys; forgejo-bootstrap holds the Forgejo token, which the scenario scripts now reuse instead of minting one per run. Both carry app.kubernetes.io/managed-by=setup.sh, and Flux prunes only what it applied.
  • Validation. validate-stack.sh checks the Flagsmith pod, liveness through the ingress, both environments against their keys, that the server hands the dashboard the seeded key, the Prometheus target, the flagsmith and monitoring-flux Kustomizations, and the migration Job.
  • Host ports bound to loopback. k3d's ingress and API ports and the registry port were published with no host address, so the podman machine's forwarder put every service on every interface of the host. Three lines in setup.sh and build.sh fix it, and validate-stack.sh requires each port to refuse the machine's own network address while answering on 127.0.0.1.
  • Second commit. Every migration Job had ttlSecondsAfterFinished: 600, and Flux re-created each one after Kubernetes deleted it, so the migrations re-ran every ten-odd minutes. The TTL is gone from all four Jobs; completed Jobs stay, and the two scripts no longer pass a missing Job as garbage-collected.
  • Also. The proto code is regenerated for the pinned protoc-gen-go (two header lines), and the PR template's headings now match what every PR here uses.

How it was tested

  • Go, Rust, and Python tests and make lint pass.
  • Clean cycle on the final code: make down, make up (9m20s, of which the seed's Flagsmith restart took 29 s), make validate, the three scenarios applied and reset with --verify. 83 assertions passed, none failed, one skipped (no grpcurl); only Watchdog firing; 328Mi and 9m idle. The first make validate after bring-up caught RustInventoryHighLatency on one slow request at idle, the p99 sensitivity already on the follow-up list; it cleared and validate passed on the re-run.
  • All three migration Jobs kept their creation timestamps and one pod each through thirty minutes and scenario 2. With the TTL, a Job re-ran inside thirteen minutes.
  • GET /config/project-overrides on the live stack serves the seeded key and the loopback URL, the flags request with that key answers 200, and the dashboard shows the login page with its flags request going to flagsmith.localhost in the browser's network tab.
  • lsof shows gvproxy 127.0.0.1:{8090,6551,5111}, and the same ports on the LAN address do not answer. Before the change they were *:port and answered 200.

Security review

  • The committed secret.yaml holds a demo database URL and a Django secret key that reads as one (demo-only-django-secret-key-not-a-real-secret), following k8s/apps/base/secrets.yaml. A real deployment generates its own.
  • The two bootstrap Secrets hold admin tokens. Anyone with get secret in the namespace can already read example-secrets and log in with the published password, and the ports those tokens sit behind are now loopback-only.
  • flagsmith.localhost is HTTP and unauthenticated at the edge, like Forgejo and Grafana here. Signup is open, and ALLOW_ADMIN_INITIATION_VIA_URL is on until the first user exists, which the seed creates during bring-up. A stack exposed beyond a laptop should set PREVENT_SIGNUP=true and ALLOW_ADMIN_INITIATION_VIA_URL=false.
  • Outbound: the telemetry heartbeat is off and the dashboard's flag fetch stays on loopback, except during the minutes of bring-up before the seed restarts the server. The only new egress is the image pull from docker.flagsmith.com, which the readme names.
  • DJANGO_ALLOWED_HOSTS=*: the hostname is not fixed (--ingress-port, port-forwards, the Service name), and the links Flagsmith puts in emails come from FLAGSMITH_DOMAIN, not the Host header. The chart hardcodes the same.
  • The bootstrap account is a superuser, matching the Forgejo bootstrap admin. ENABLE_ADMIN_ACCESS_USER_PASS is unset.

Flagsmith joins the stack as infrastructure: deployed by Flux from
k8s/infra/flagsmith like the monitoring stack, seeded by setup.sh like
Forgejo, checked by validate-stack.sh, and documented with the rest. It
is inert at baseline. No flags are defined and no service reads one;
later changes wire the SDKs and add scenarios.

The manifests are a namespace, a one-replica Deployment of the unified
image pinned by tag and digest, a Service, an Ingress on
flagsmith.localhost, a demo-only Secret, a migrate-flagsmith-v1 Job,
and a ServiceMonitor. The container runs serve, so migrations run only
in the Job, and readiness fails until the Job has applied the schema.
Gunicorn runs one worker. There is no CPU limit, like every other
Deployment here. The flagsmith database joins orders in the PostgreSQL
initdb ConfigMap. The Flux Kustomization depends on apps, which covers
both the operator CRDs and the database.

Two Flagsmith defaults that reach the internet are off. ENABLE_TELEMETRY
posts object counts to api.flagsmith.com from Django's post_migrate
signal. The dashboard fetches its own feature flags from Flagsmith's
hosted service on every page load: the two settings default to None,
the view that builds the frontend config omits a None, and the
self-hosted bundle hardcodes a key and edge.api.flagsmith.com, so empty
strings replace both. Everything else it could call out to is already
off at its default.

setup.sh seeds a bootstrap superuser through the signup endpoint, an
organization and project named example-stack, and exactly the staging
and production environments, with --flagsmith-password for the account.
Later runs log in with the password rather than resetting it. The admin
token and the two environment keys go into a flagsmith-bootstrap Secret,
and the Forgejo token setup.sh already mints goes into forgejo-bootstrap,
which the scenario scripts now reuse instead of minting one per run.
Both are bootstrap state Flux does not own and does not prune.

Every host port the stack opens is bound to 127.0.0.1. k3d's ingress
and API ports and the registry port were published with no host
address, and the podman machine's forwarder put them on every interface
of the host, so a running cluster was reachable from whatever network
the machine was on with the demo credentials this repository publishes.
validate-stack.sh now asks the routing table for the machine's own
address and requires each port to refuse there while answering on
127.0.0.1. It also checks the Flagsmith pod, liveness through the
ingress, both seeded environments against their keys, the flagsmith
Prometheus target, the flagsmith and monitoring-flux Kustomizations,
and the migration Job. Two checks that appended a second 000 to curl's
own now read curl's value.

The proto code is regenerated for the protoc-gen-go go.mod pins; the
diff is the two header lines. The pull request template uses the three
headings every pull request here has actually been written under.

Assisted-by: Claude:claude-opus-5 [claude-code]
Assisted-by: Claude:claude-fable-5-1 [claude-code]
Signed-off-by: Andrew McCabe <andy@endgrainlabs.com>
Each migration Job had ttlSecondsAfterFinished: 600. Kubernetes deleted
the completed Job after ten minutes, Flux re-created it on the next
reconcile, and the migrations ran on a loop for as long as the cluster
was up. The TTL is gone from all four Jobs; completed Jobs stay, and the
versioned names already handle a change. validate-stack.sh and
smoke-test.sh no longer pass a missing Job as garbage-collected.

docs/topology.md takes back one sentence: Flagsmith does exit at startup
when it comes up ahead of its Job and finds no schema.

Assisted-by: Claude:claude-fable-5-1 [claude-code]
Signed-off-by: Andrew McCabe <andy@endgrainlabs.com>
Blanking FLAGSMITH_ON_FLAGSMITH_API_KEY and _API_URL stopped the dashboard's
call to Flagsmith's hosted service, but the frontend's only failure handler for
its own flags renders the Maintenance page, so the dashboard was unusable.

Follow Flagsmith's "Running Flagsmith on Flagsmith" docs instead. setup.sh
finds or creates a flagsmith-dashboard project with one environment, writes
its client-side key to a flagsmith-dashboard Secret in the flagsmith
namespace, and restarts Flagsmith when the key changed. The Deployment reads
the key from that Secret, optionally, and points the API URL at the stack's
ingress, which setup.sh already rewrites for --ingress-port. validate-stack.sh
checks that /config/project-overrides serves the key.

Assisted-by: Claude:claude-opus-5 [claude-code]
Signed-off-by: Andrew McCabe <andy@endgrainlabs.com>
@amccabe
amccabe merged commit 18127ef into main Sep 23, 2026
10 checks passed
@amccabe
amccabe deleted the feat/flagsmith branch September 23, 2026 05:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant