diff --git a/README.md b/README.md index c803279..a5660dd 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ is self-contained after startup (see [Prerequisites](#prerequisites) and environment and refreshed every ten seconds; see [docs/services.md](docs/services.md#feature-flags) - A local image registry that the cluster pulls from. -- Three failure scenarios that break the stack in a specific way and revert. +- Four failure scenarios that break the stack in a specific way and revert. This is how they are organized in the repository @@ -265,15 +265,17 @@ kubectl -n forgejo exec deploy/forgejo -- \ ## Failure scenarios -Each failure scenario pushes a kustomize overlay to Forgejo, which Flux -then reconciles. Each scenario is reverted by running its associated script -with `--reset`. +Scenarios 1 to 3 push a kustomize overlay to Forgejo, which Flux then +reconciles. Scenario 4 turns feature flags on through Flagsmith's API and +changes nothing in Git. Each scenario is reverted by running its associated +script with `--reset`. | Scenario | Description | |---|---| | broken service port (1) | The `go-grpc` Service targets the wrong port. The pod stays healthy, the traffic is dropped, `go-api` order creation returns 502. | | bad migration (2) | A migration renames a column. The Job succeeds, and `rust-inventory` queries start returning 500. | | invalid pricing assumption (3) | `go-grpc` returns a different currency for one warehouse. Both services are healthy, and orders for that warehouse fail with 422. | +| feature flag triple (4) | Three Flagsmith flags, one per service, each inert alone, together price the `west` warehouse item in EUR, and its orders fail with 422. Forgejo and Flux show no change, and no alert fires. | To run scenario 1 and revert to the baseline state, you'd execute ```sh diff --git a/docs/scenarios.md b/docs/scenarios.md index 3272b4a..ff99e7d 100644 --- a/docs/scenarios.md +++ b/docs/scenarios.md @@ -1,24 +1,32 @@ # Failure scenarios -There are three supported reproducible failures, each representing a -different class of distributed system problem. Each is applied through -Flux and reverted the same way, with the script undoing anything the -scenario changed outside Git. +There are four supported reproducible failures, each representing a +different class of distributed system problem. The first three are applied +through Flux and reverted the same way, with the script undoing anything +the scenario changed outside Git. The fourth changes nothing in Git: it +turns feature flags on through Flagsmith's API and off again the same way. The stack must be running first by executing `make up` or `bash scripts/setup.sh`. Running `demo.sh` for each scenario is idempotent. ## How to apply a scenario -Each scenario directory has an `overlay/` directory (a kustomize overlay -over `k8s/apps/base`) and a `demo.sh` script to drive it. `demo.sh` clones +Each scenario directory has a `demo.sh` script to drive it. Scenarios 1 +to 3 also have an `overlay/` directory (a kustomize overlay over +`k8s/apps/base`), and their `demo.sh` clones the in-cluster Forgejo repository, commits the overlay, pushes it, and patches the Flux `apps` Kustomization to reconcile from the overlay path. A webhook triggers a Flux reconciliation immediately. +Scenario 4 has no overlay. Its `demo.sh` reads the Flagsmith admin token +and the production environment key from the `flagsmith-bootstrap` Secret +and switches flags through the Flagsmith API at +`http://flagsmith.localhost:8090/api/v1`. + `scenarios/lib.sh` has shared code between the scenarios' `demo.sh` scripts. They share flags, the Forgejo token and port-forward, the clone, the overlay -push, the Flux path switch, and a wait helper. The token is read from the +push, the Flux path switch, the Flagsmith API call and flag switch, and a +wait helper. The Forgejo token is read from the `forgejo-bootstrap` Secret the bring-up wrote and reused while Forgejo still accepts it, which is the "Reusing the Forgejo access token" line the scripts print; one is minted through the Forgejo command line when that Secret is @@ -29,8 +37,11 @@ in the clone, so a file dropped from a scenario does not survive in Forgejo and re-running the script with nothing to change pushes nothing. A wait that times out exits non-zero. -Running `demo.sh` with `--reset` points the Kustomization path back at -`./k8s/apps/base` and undoes whatever else the scenario changed outside Git. +Running `demo.sh` with `--reset` undoes its own scenario. For scenarios 1 +to 3 it points the Kustomization path back at `./k8s/apps/base` and undoes +whatever else the scenario changed outside Git. For scenario 4 it turns the +three flags off. One scenario is applied and verified at a time, and a +combination of scenarios is a scenario of its own. ## Verifying a scenario @@ -183,6 +194,98 @@ an order for Gadget returning 422 with a body naming the currency, and `go-grpc` health returning `SERVING`. After a reset it asserts that an order for Gadget returns 201. +## Scenario 4: feature flag triple + +This is an emergent interaction between three feature flags, each owned by +a different service. + +The scenario models three feature flags that cause a functional regression +only when all three are enabled. Each service reads one flag from the +`production` environment of the `example-stack` project in Flagsmith, all +three off at baseline, as [services.md](services.md#feature-flags) +describes. The demo enables them in this sequence, through Flagsmith's API. + +1. `inventory.expose_region`: `rust-inventory` adds a `region` to each + item, `us-east` for the `east` warehouse and `eu-west` for `west`. +2. `orders.forward_region`: `go-api` passes the item's region to pricing. +3. `pricing.regional_currency`: `go-grpc` prices `eu-west` in EUR. + +With all three on, an order for Gadget, the `west` warehouse item, is +priced in EUR and `go-api` answers 422, the symptom scenario 3 produces +through `REGIONAL_PRICING`, from a different cause. Widget and Sprocket are +`east`, whose region `us-east` has no currency mapping, and still succeed in +USD. The services refresh their flags every ten seconds, so each step shows +within about ten seconds. + +Any two flags are inert. + +- `inventory.expose_region` and `orders.forward_region` on, without + `pricing.regional_currency`: the region reaches `go-grpc`, which ignores + it. +- `inventory.expose_region` and `pricing.regional_currency` on, without + `orders.forward_region`: `go-api` never forwards the region. +- `orders.forward_region` and `pricing.regional_currency` on, without + `inventory.expose_region`: there is no region to forward. + +These are properties of the code, and each service's gating has a unit +test. The other orderings therefore end the same way as 1, 2, 3, and the +demo does not walk them: it verifies the one path it walks. + +What a careful observer sees + +- All three services healthy, every readiness check passing. +- `rust-inventory` items carrying a `region`. +- `go-api` orders for Widget and Sprocket succeeding in USD. +- `go-api` orders for Gadget returning 422 with `unsupported currency: EUR`. +- `goapi_http_requests_total{status="422"}` rising, and no alert firing: the + `goapi:http_error_rate` recording rule matches 5xx only. +- No commit in Forgejo and no change to any Flux Kustomization. +- No entry in Flagsmith's audit log. The stack runs Flagsmith on its free + plan, whose audit log visibility is zero days, so the audit log is empty + on this stack. +- In the Flagsmith dashboard at `http://flagsmith.localhost:8090/`, each + flag's current state in the `production` environment. That is the only + record of the change, with no history and no author. + +No single change breaks anything, and each one can be checked on its own and +found harmless. The failure exists only in the combination, across three +services and three owners. It arrives outside Git, so the deploy history +shows nothing. It is a 4xx, so the error-rate rules do not see it. Finding +it means noticing that the 422s began without a deploy, then reading the +current state of every flag and working out what they do together. + +To run this scenario: + +```sh +bash scenarios/scenario-4-feature-flag-triple/demo.sh --verify +bash scenarios/scenario-4-feature-flag-triple/demo.sh --reset --verify +``` + +An apply starts by restoring the baseline, turning off any of the three +flags that is on, and says whether it was already there. It then enables +the flags in the sequence 1, 2, 3. Without `--verify` it pauses briefly +between them and prints what to look at. With `--verify` it also asserts +between the steps. After each of the first two flags it +asserts that orders for Widget and Gadget return 201 in USD, that the +Gadget item carries `"region":"eu-west"`, and that `go-api` and +`rust-inventory` readiness return 200. After the third it asserts that an +order for Gadget returns 422 with a body naming the currency, an order for +Widget returns 201 in USD, and +`goapi_http_requests_total{status="422"}` rises from a reading taken before +the third flag went on. + +A second apply therefore restores the baseline and replays the same walk, +ending in the same place. + +Reset restores the baseline and verifies it, nothing else. It says whether +the flags were already at baseline, then turns all three off in reverse +order, `pricing.regional_currency`, `orders.forward_region`, +`inventory.expose_region`, whatever state they were in, leaving any flag +already off alone. With `--verify` it asserts the baseline: orders for +Widget and Gadget return 201 in USD, the Gadget item carries no `region`, +and `go-api` and `rust-inventory` readiness return 200. A second reset +passes. + ## Alerts take time to clear The alert rules use a five minute rate window with a two minute `for` clause. diff --git a/docs/services.md b/docs/services.md index b6411b3..0127759 100644 --- a/docs/services.md +++ b/docs/services.md @@ -323,7 +323,8 @@ request names one. With all three on, an order for Gadget, the `west` warehouse item, is priced in EUR and `go-api` answers 422, the symptom scenario 3 produces through `REGIONAL_PRICING`, from a different cause. Orders for Widget and Sprocket, both `east`, still succeed. With any -flag off every order is priced in USD. +flag off every order is priced in USD. [Scenario 4](scenarios.md#scenario-4-feature-flag-triple) +enables the three flags in sequence and checks the stack after each change. `go-api` and `go-grpc` use Flagsmith's server-side Go SDK, `flagsmith-go-client`, wrapped once in `internal/flags`, in local diff --git a/docs/topology.md b/docs/topology.md index e59e950..0c2d0c4 100644 --- a/docs/topology.md +++ b/docs/topology.md @@ -138,10 +138,12 @@ declared in `infra` alone, so nothing a scenario does to `apps` can prune it. Flagsmith's migration Job starts in the same reconcile as PostgreSQL and retries until the database answers. -A failure scenario works in the same way. It clones the Forgejo +Failure scenarios 1 to 3 work in the same way. Each clones the Forgejo repository, commits an overlay, pushes, and patches the `apps` Kustomization to point at the overlay path. Reverting points the path -back at `./k8s/apps/base`. +back at `./k8s/apps/base`. Scenario 4 goes around this path: it turns +feature flags on through Flagsmith's API, so Forgejo and Flux see no +change. ## The monitoring stack diff --git a/k8s/apps/base/landing.yaml b/k8s/apps/base/landing.yaml index 4e7ddbc..802c6f9 100644 --- a/k8s/apps/base/landing.yaml +++ b/k8s/apps/base/landing.yaml @@ -56,11 +56,12 @@ data:

curl -H "Authorization: Bearer dev-token" http://rust.localhost:8090/api/v1/inventory

Failure scenarios

-

Each scenario pushes a kustomize overlay to Forgejo, Flux reconciles it, and --reset reverts it. Run them from a checkout of the repository.

+

Scenarios 1 to 3 push a kustomize overlay to Forgejo and Flux reconciles it; scenario 4 turns feature flags on through Flagsmith's API. --reset reverts each. Run them from a checkout of the repository.

Scripts

diff --git a/scenarios/lib.sh b/scenarios/lib.sh index d450a86..f431208 100644 --- a/scenarios/lib.sh +++ b/scenarios/lib.sh @@ -1,6 +1,7 @@ # shellcheck shell=bash # Shared machinery for the scenario scripts: flags, Forgejo access, the clone -# of the in-cluster repository, the Flux path switch, and the wait helper. +# of the in-cluster repository, the Flux path switch, Flagsmith access and +# the flag switch, and the wait helper. # # Sourced by each demo.sh, never run on its own. A demo.sh sources this file, # calls scenario_describe, and then scenario_parse_args "$@". A demo.sh that @@ -27,6 +28,13 @@ FORGEJO_ORG="endgrainlabs" FORGEJO_REPO="example-stack" FORGEJO_ADMIN_USER="bootstrap" FLUX_NS="flux-system" +# The Flagsmith project setup.sh seeds, which holds the flags the services +# read. Its production environment is the one they read. +FLAGSMITH_PROJECT="example-stack" +# The three flags, one per service, all off at baseline. +FLAG_INVENTORY="inventory.expose_region" +FLAG_ORDERS="orders.forward_region" +FLAG_PRICING="pricing.regional_currency" # Local ports for the port-forwards: ports unlikely to be in use on the host. FORGEJO_PF_PORT="13000" GRPC_PF_PORT="19092" @@ -201,6 +209,190 @@ scenario_set_flux_path() { reconcile.fluxcd.io/requestedAt="$(date +%s)" } +# Answers a question about a Flagsmith API response. A copy of flagsmith_fact +# in scripts/setup.sh, which this file does not source; the modes are the +# same: +# +# field KEY one key of an object +# find NAME KEY one key of the first list entry whose name is NAME +# first KEY one key of the first list entry +# names the name of every list entry, one per line +# +# A list answer is either a bare array or a paginated object with "results". +# A boolean prints as true or false, the way the JSON spells it. Nothing found +# is a non-zero exit, so callers fall back with || echo "". +scenario_flagsmith_fact() { + local response="$1" + shift + printf '%s' "${response}" | python3 -c ' +import json +import sys + +mode = sys.argv[1] +try: + doc = json.load(sys.stdin) +except ValueError: + sys.exit(1) + + +def emit(value): + if value is None: + sys.exit(1) + print(json.dumps(value) if isinstance(value, bool) else value) + sys.exit(0) + + +if mode == "field": + emit(doc.get(sys.argv[2]) if isinstance(doc, dict) else None) + +items = doc.get("results", []) if isinstance(doc, dict) else doc +if mode == "names": + for item in items: + print(item["name"]) + sys.exit(0) + +if mode == "first": + emit(items[0].get(sys.argv[2]) if items else None) + +for item in items: + if item["name"] == sys.argv[2]: + emit(item[sys.argv[3]]) +sys.exit(1) +' "$@" 2>/dev/null +} + +# A call to the Flagsmith admin API through the ingress, with the admin token +# attached. A body is passed in a variable built beforehand: bash 3.2 +# brace-expands JSON written inside "$(...)". +scenario_flagsmith_api() { + local method="$1" + local path="$2" + local body="${3:-}" + local api="http://flagsmith.localhost:${INGRESS_PORT}/api/v1" + if [ -n "${body}" ]; then + curl -s --max-time 15 -X "${method}" \ + -H "Authorization: Token ${FLAGSMITH_TOKEN}" \ + -H "Content-Type: application/json" \ + -d "${body}" \ + "${api}${path}" || echo "" + else + curl -s --max-time 15 -X "${method}" \ + -H "Authorization: Token ${FLAGSMITH_TOKEN}" \ + "${api}${path}" || echo "" + fi +} + +# Reads the admin token and the production client key that setup.sh stored +# in the flagsmith-bootstrap Secret, and finds the seeded project. Flagsmith +# generates both at seed time, so the Secret is the only place they are kept. +scenario_open_flagsmith() { + local projects + + echo "==> Reading the Flagsmith admin token and production key" + FLAGSMITH_TOKEN=$(kubectl -n "${NAMESPACE}" get secret flagsmith-bootstrap \ + -o jsonpath='{.data.admin-token}' 2>/dev/null | base64 --decode 2>/dev/null || echo "") + FLAGSMITH_PRODUCTION_KEY=$(kubectl -n "${NAMESPACE}" get secret flagsmith-bootstrap \ + -o jsonpath='{.data.production}' 2>/dev/null | base64 --decode 2>/dev/null || echo "") + if [ -z "${FLAGSMITH_TOKEN}" ] || [ -z "${FLAGSMITH_PRODUCTION_KEY}" ]; then + echo "ERROR: the flagsmith-bootstrap Secret in ${NAMESPACE} has no admin-token or production key;" >&2 + echo " run scripts/setup.sh to seed Flagsmith" >&2 + exit 1 + fi + + projects=$(scenario_flagsmith_api GET /projects/) + FLAGSMITH_PROJECT_ID=$(scenario_flagsmith_fact "${projects}" find "${FLAGSMITH_PROJECT}" id || echo "") + if [ -z "${FLAGSMITH_PROJECT_ID}" ]; then + echo "ERROR: could not find the Flagsmith project '${FLAGSMITH_PROJECT}'" >&2 + echo " Flagsmith answered: ${projects}" >&2 + exit 1 + fi +} + +# Finds a feature's state in the production environment, the one the services +# read, and sets FLAG_STATE_ID and FLAG_STATE_ENABLED (true or false). It sets +# globals rather than printing, so a caller runs it directly and not in a +# subshell. Needs scenario_open_flagsmith first. +FLAG_STATE_ID="" +FLAG_STATE_ENABLED="" +scenario_flag_state() { + local name="$1" + local features feature_id states + + features=$(scenario_flagsmith_api GET "/projects/${FLAGSMITH_PROJECT_ID}/features/") + feature_id=$(scenario_flagsmith_fact "${features}" find "${name}" id || echo "") + if [ -z "${feature_id}" ]; then + echo "ERROR: Flagsmith has no feature '${name}' in the project '${FLAGSMITH_PROJECT}'" >&2 + echo " Flagsmith answered: ${features}" >&2 + exit 1 + fi + + # The environment's own state for the feature, not a segment's or an + # identity's: this endpoint lists only those. + states=$(scenario_flagsmith_api GET "/environments/${FLAGSMITH_PRODUCTION_KEY}/featurestates/?feature=${feature_id}") + FLAG_STATE_ID=$(scenario_flagsmith_fact "${states}" first id || echo "") + FLAG_STATE_ENABLED=$(scenario_flagsmith_fact "${states}" first enabled || echo "") + if [ -z "${FLAG_STATE_ID}" ]; then + echo "ERROR: Flagsmith has no production state for feature '${name}'" >&2 + echo " Flagsmith answered: ${states}" >&2 + exit 1 + fi +} + +# Turns one feature's enabled bit on or off in the production environment. +# Only that bit: Flagsmith keeps a feature's value, which can be a string, a +# number, or JSON, apart from whether it is enabled, and this leaves the value +# alone. A feature already in that state is left alone too, so a re-run +# changes nothing. +scenario_set_flag_enabled() { + local name="$1" + local state="$2" + local enabled body response + + case "${state}" in + on) enabled="true" ;; + off) enabled="false" ;; + *) + echo "ERROR: scenario_set_flag_enabled takes on or off, not '${state}'" >&2 + exit 1 ;; + esac + + scenario_flag_state "${name}" + if [ "${FLAG_STATE_ENABLED}" = "${enabled}" ]; then + echo " '${name}' is already ${state} in production" + return 0 + fi + + body='{"enabled":'"${enabled}"'}' + response=$(scenario_flagsmith_api PUT "/environments/${FLAGSMITH_PRODUCTION_KEY}/featurestates/${FLAG_STATE_ID}/" "${body}") + if [ "$(scenario_flagsmith_fact "${response}" field enabled || echo "")" != "${enabled}" ]; then + echo "ERROR: could not turn feature '${name}' ${state} in production" >&2 + echo " Flagsmith answered: ${response}" >&2 + exit 1 + fi + echo " '${name}' turned ${state} in production" +} + +# Exits zero when all three flags are off in production, the baseline, and +# prints nothing: the callers say what the answer means for them. +scenario_flags_at_baseline() { + local name + for name in "${FLAG_PRICING}" "${FLAG_ORDERS}" "${FLAG_INVENTORY}"; do + scenario_flag_state "${name}" + if [ "${FLAG_STATE_ENABLED}" != "false" ]; then + return 1 + fi + done + return 0 +} + +# Turns all three flags off, in the reverse of the order scenario 4 turns +# them on. +scenario_flags_restore_baseline() { + scenario_set_flag_enabled "${FLAG_PRICING}" off + scenario_set_flag_enabled "${FLAG_ORDERS}" off + scenario_set_flag_enabled "${FLAG_INVENTORY}" off +} + # Polls a command until it prints the expected value, for up to two minutes. # A timeout is an error: the rest of a demo.sh describes a state the cluster # would not be in. diff --git a/scenarios/scenario-4-feature-flag-triple/demo.sh b/scenarios/scenario-4-feature-flag-triple/demo.sh new file mode 100644 index 0000000..95c1637 --- /dev/null +++ b/scenarios/scenario-4-feature-flag-triple/demo.sh @@ -0,0 +1,230 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Scenario 4: feature flag triple +# +# Models three feature flags in Flagsmith's production environment, one per +# service, that cause a functional regression only when all three are +# enabled: inventory.expose_region (rust-inventory adds a region derived from +# the warehouse), orders.forward_region (go-api passes that region to +# pricing), and pricing.regional_currency (go-grpc prices eu-west in EUR). +# With all three on, Gadget (the west warehouse item) is priced in EUR and +# go-api rejects the order with 422; Widget (east) keeps succeeding in USD. +# +# The demo enables them in the sequence 1, 2, 3 and, with --verify, checks +# the stack after each change. Any two flags are inert, so the other +# orderings end the same way; the demo does not walk them. An apply first restores the +# baseline, all three off, so a second apply replays the same walk. +# +# The change goes through Flagsmith's API, not Git: nothing is pushed to +# Forgejo and the Flux path is not touched. It leaves no entry in Flagsmith's +# audit log on this stack, whose free plan shows zero days of it. The flags' +# current state in the dashboard is the only record, with no history and no +# author. The failure is a 4xx, which no alert rule watches. +# +# Invoke with: bash scenarios/scenario-4-feature-flag-triple/demo.sh [--reset] + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# shellcheck source=../lib.sh disable=SC1091 +source "${SCRIPT_DIR}/../lib.sh" + +scenario_describe "scenario-4-feature-flag-triple" \ + "Three Flagsmith flags, one per service, cause a functional regression +only when all three are enabled: the west warehouse item is priced in EUR, +which go-api rejects with 422. The demo enables them in the sequence 1, 2, 3 +and with --verify checks the stack between each change; any two flags are +inert, so the other orderings end the same way, and the demo does not walk +them. The change goes through Flagsmith's API: Forgejo and Flux show +nothing, Flagsmith's audit log on this stack shows no entry (the free plan's +audit visibility is zero days), and the dashboard shows only each flag's +current state, with no history and no author. No alert fires, because the +failure is a 4xx." +scenario_parse_args "$@" + +# The services refresh their flags every ten seconds, so a step shows within +# this many seconds or it is not going to. +STEP_TIMEOUT=30 +# Without --verify, the gap between flags, so each change is a separate +# moment in the logs and the dashboard. +STEP_PAUSE=2 + +# go-api is not restarted, so the sum over its pods keeps counting from the +# baseline taken just before the third flag goes on. +ERRORS_422_QUERY='sum(goapi_http_requests_total{status="422"}) or vector(0)' +ERRORS_422_BEFORE="" + +# The region rust-inventory gives an item: the value, "none" when the field +# is absent, or the HTTP status when the request did not return an item. +inventory_region() { + local body_file status region + + body_file=$(mktemp) + status=$(curl -s -o "${body_file}" -w '%{http_code}' --max-time 15 \ + -H "Authorization: Bearer ${API_TOKEN}" \ + "http://rust.localhost:${INGRESS_PORT}/api/v1/inventory/$1" || echo "000") + region=$(python3 -c 'import json,sys; print(json.load(sys.stdin).get("region", "none"))' \ + <"${body_file}" 2>/dev/null || echo "") + rm -f "${body_file}" + + if [ "${status}" = "200" ] && [ -n "${region}" ]; then + printf '%s' "${region}" + else + printf 'status %s' "${status}" + fi +} + +# An order for the item succeeds, and in USD. A polling assertion runs its +# command in a subshell, so one more order fills ORDER_BODY for the currency. +assert_order_succeeds() { + local label="$1" + local item_id="$2" + + scenario_assert_eventually "order for ${label} returns 201" "201" "${STEP_TIMEOUT}" \ + scenario_order_status "${item_id}" 1 + scenario_order "${item_id}" 1 + scenario_check "the ${label} order is in USD" "yes" \ + "$(scenario_order_body_has '"currency":"USD"')" +} + +# The stack is correct as a whole: both warehouses' items order in USD, both +# services are ready, and the Gadget item carries the region it should. +assert_stack_correct() { + local gadget_region="$1" + + assert_order_succeeds "Widget" "${ITEM_EAST}" + assert_order_succeeds "Gadget" "${ITEM_WEST}" + scenario_assert_eventually "Gadget item region is ${gadget_region}" "${gadget_region}" \ + "${STEP_TIMEOUT}" inventory_region "${ITEM_WEST}" + scenario_assert_eventually "go-api /readyz returns 200" "200" "${STEP_TIMEOUT}" \ + scenario_get_status "http://goapi.localhost:${INGRESS_PORT}/readyz" + scenario_assert_eventually "rust-inventory /readyz returns 200" "200" "${STEP_TIMEOUT}" \ + scenario_get_status "http://rust.localhost:${INGRESS_PORT}/readyz" +} + +scenario_verify_break() { + scenario_assert_eventually "order for Gadget returns 422" "422" "${STEP_TIMEOUT}" \ + scenario_order_status "${ITEM_WEST}" 1 + scenario_order "${ITEM_WEST}" 1 + scenario_check "the 422 names the currency" "yes" \ + "$(scenario_order_body_has "unsupported currency: EUR")" + assert_order_succeeds "Widget" "${ITEM_EAST}" + scenario_assert_increased "goapi_http_requests_total{status=\"422\"} rising" \ + "${ERRORS_422_QUERY}" "${ERRORS_422_BEFORE}" 150 \ + "scenario_order_status ${ITEM_WEST} 1" +} + +scenario_verify_reset() { + assert_stack_correct "none" +} + +scenario_open_flagsmith + +# The assertions between steps run before scenario_verify, which is what +# normally reads the token. +if [ "${VERIFY}" -eq 1 ]; then + API_TOKEN=$(scenario_api_token) +fi + +if [ "${ACTION}" = "break" ]; then + echo "==> Injecting scenario 4: three feature flags that only fail together" + + # The walk starts from the baseline, so a second apply replays it rather + # than finding every flag on and checking a first-apply world. + echo "==> Restoring the flag baseline before the walk" + if scenario_flags_at_baseline; then + echo " All three flags are already off" + else + echo " Not at baseline, turning all three flags off" + scenario_flags_restore_baseline + fi + + echo "==> Enabling the flags in the sequence 1, 2, 3" + scenario_set_flag_enabled "${FLAG_INVENTORY}" on + if [ "${VERIFY}" -eq 1 ]; then + echo "==> Verifying the stack with ${FLAG_INVENTORY} on" + assert_stack_correct "eu-west" + else + sleep "${STEP_PAUSE}" + fi + + scenario_set_flag_enabled "${FLAG_ORDERS}" on + if [ "${VERIFY}" -eq 1 ]; then + echo "==> Verifying the stack with ${FLAG_INVENTORY} and ${FLAG_ORDERS} on" + assert_stack_correct "eu-west" + else + sleep "${STEP_PAUSE}" + fi + + if [ "${VERIFY}" -eq 1 ]; then + ERRORS_422_BEFORE=$(scenario_prom_query "${ERRORS_422_QUERY}") + fi + scenario_set_flag_enabled "${FLAG_PRICING}" on + + echo "" + echo "==> Scenario 4 active. The services pick the flags up within ten seconds." + echo " The regression needs all three flags enabled. The demo enabled them in" + echo " the sequence 1, 2, 3; any two are inert, so the other orderings end the" + echo " same way, and the demo does not walk them." + echo "" + echo " Expected symptoms:" + echo " - rust-inventory: items carry a region (Gadget eu-west, Widget us-east)" + echo " - go-api + Widget (item 001, east, us-east): orders succeed (USD pricing)" + echo " - go-api + Gadget (item 002, west, eu-west): 422 unsupported currency: EUR" + echo " - all three services healthy, every readiness check passing" + echo " - Prometheus: goapi_http_requests_total with status=422 rises" + echo " - Alerts expected to fire: none." + echo " The goapi:http_error_rate recording rule matches {status=~\"5..\"} only," + echo " so the 422s do not raise it and GoApiHighErrorRate stays quiet." + echo "" + echo " Where the change shows:" + echo " - Forgejo: no commit. Flux: no change to any Kustomization." + echo " - Flagsmith's audit log: no entry on this stack; the free plan's audit" + echo " visibility is zero days." + echo " - Flagsmith's dashboard, the only record: each flag's current state in" + echo " the production environment of the example-stack project, with no" + echo " history and no author:" + echo " http://flagsmith.localhost:${INGRESS_PORT}/" + echo "" + echo " Try (succeeds):" + echo " curl -s -H 'Authorization: Bearer dev-token' -H 'Content-Type: application/json' \\" + echo " -d '{\"item_id\":\"a0000000-0000-0000-0000-000000000001\",\"quantity\":1}' \\" + echo " http://goapi.localhost:${INGRESS_PORT}/api/v1/orders" + echo "" + echo " Try (fails with 422):" + echo " curl -s -H 'Authorization: Bearer dev-token' -H 'Content-Type: application/json' \\" + echo " -d '{\"item_id\":\"a0000000-0000-0000-0000-000000000002\",\"quantity\":1}' \\" + echo " http://goapi.localhost:${INGRESS_PORT}/api/v1/orders" + echo "" + echo " See the region the first flag added:" + echo " curl -s -H 'Authorization: Bearer dev-token' \\" + echo " http://rust.localhost:${INGRESS_PORT}/api/v1/inventory/a0000000-0000-0000-0000-000000000002" + echo "" + echo " Check alert state (expect no new firing alerts):" + echo " curl -s http://prometheus.localhost:${INGRESS_PORT}/api/v1/alerts | python3 -c \\" + echo " 'import json,sys; [print(a[\"labels\"][\"alertname\"],a[\"state\"]) for a in json.load(sys.stdin)[\"data\"][\"alerts\"]]'" + echo "" + echo " Reset: bash scenarios/${SCENARIO_ID}/demo.sh --reset" + +elif [ "${ACTION}" = "reset" ]; then + # Reset restores the baseline from whatever state the flags are in and + # checks only the baseline: the states on the way down belong to no + # scenario. + echo "==> Clearing scenario 4: restoring the flag baseline" + if scenario_flags_at_baseline; then + echo " All three flags are already off, the stack is at baseline" + else + echo " Not at baseline, turning all three flags off" + fi + scenario_flags_restore_baseline + + echo "==> Scenario 4 cleared. The services pick the flags up within ten seconds." + echo "" + echo " Note: this scenario fires none of the existing alerts, because" + echo " goapi:http_error_rate watches 5xx only, so there is nothing to" + echo " wait on. If alerts from a prior scenario are still settling," + echo " validate-stack.sh reports them as firing until the rate([5m])" + echo " window and the for:2m clause have passed." +fi + +scenario_verify