Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .claude/skills/agentic-delivery/INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ Generated by `scripts/skill_index.py` (do not hand-edit; rerun it after changing
| `references/merge-queue-worktrees.md` | draining a ready queue or a single merge seat, landing a train (merge_train.py, train_land.sh, land… | 11204 |
| `references/multi-session-coordination.md` | two or more independent agent sessions — separate processes, machines, or accounts — coordinate on… | 18475 |
| `references/operating-discipline.md` | setting up a new machine or a new fleet, or auditing whether the always-on operating layer actually… | 2902 |
| `references/pipeline-optimizer.md` | a delivery pipeline is slow (a commit or push hook over about a minute, CI over about ten minutes,… | 4838 |
| `references/pipeline-optimizer.md` | a delivery pipeline is slow (a commit or push hook over about a minute, CI over about ten minutes,… | 4870 |
| `references/production-grade-product-playbook.md` | you are shaping a user-facing product surface in G3 Design, so you build to the domain-P quality ba… | 1658 |
| `references/project-state.md` | beginning multi-step delivery, changing objectives, transferring ownership, checkpointing, or recov… | 1885 |
| `references/red-gate-triage.md` | a gate, union, or CI check is red and you must decide whether it is real: attributing a red union (… | 6796 |
| `references/retrospective.md` | running G10, or when a target's own incident/postmortem practice is in scope for a review. | 1211 |
| `references/red-gate-triage.md` | a gate, union, or CI check is red and you must decide whether it is real: attributing a red union (… | 6904 |
| `references/retrospective.md` | running G10, or when a target's own incident/postmortem practice is in scope for a review. | 1273 |
| `references/roles.md` | standing up (or running) the full delivery team on a repo: who does what, when a hat fires, which g… | 6830 |
| `references/support-ops.md` | the product has users who file tickets, bug reports, or feature requests, and you need a support sy… | 1113 |
| `references/template-adr.md` | G3 Design must record a decision — an interface, NFR budget, or data/security choice — as an ADR (c… | 660 |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ This agrees with `host-enforcement.md` *Cost discipline* (run every gate locally
4. **Escalate a blocker once.** A permission, sandbox, branch-protection or secret gap goes to the owner as ONE batched ask with the exact commands to run. Then continue with the next row that needs none of it. Do not retry the blocked step or route around the control (`SKILL.md` *Human gates*).
5. **A data gap is not a stop.** `profile` records an API rate limit and keeps what it has; resume with the same `--data` file, or measure locally (`time`, `suites --run`, `hooks --run`).
6. **A watchdog or budget is shorter than the hard timeout it pre-empts.** Equal or longer never fires first; `scripts/test_budget_below_timeout.py` asserts it for the CI step, job and lease timeouts.
7. **Deletion budget.** Each pipeline fix removes at least one knob or code path, or says in its report line why none could go.

## Step 1: map (mandatory)

Expand Down
2 changes: 2 additions & 0 deletions .claude/skills/agentic-delivery/references/red-gate-triage.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ only when every new listed failure re-ran as a flake and the runner's own `FAILU
lines. **CANT rule:** the per-PR bisect re-runs each real id through the same flake triage (never one run), blames a PR only
when it fails the id alone, and a failure that passes alone for every PR (`FLAKY`, no blame) or is not pinned to a culprit makes the train print `CANT` and exit 3 with nothing landed. Tests: `scripts/test-train-resilience.sh`, `test-peer-pains-2.sh`, `test-train-flake-base.sh`.

**Rerun and batch rules.** Re-run a failing title solo with one worker, never with the suite's parallelism; 2 of 2 solo passes is a flake. Before any bisect, run the same title on the base at the same load (a paired base run) so load is not blamed on the PR. Size the batch with an AIMD window rather than a fixed cap: start at 4, add 1 after each green train, halve after a red one. Test: `scripts/test-train-flake-cleanup.sh`.

**Split on an unattributed red (`TRAIN_SPLIT_DEPTH`, default 2; `0` restores plain CANT).** At roughly 5 merges an hour, one red train that names no culprit used to land nothing and stall the whole queue. Now, when the per-PR bisect finds no culprit (`no culprit pinned` or `real failure(s) not pinned`, after any pinned culprits are dropped) and at least 2 PRs remain, `train_land.sh` splits them into two halves. Each half re-runs the full gate (verify plus browser, with the same flake and base-red triage) stacked on the fresh base and lands only when its own gate is green, so unverified code never lands. The left half lands first and the right half is gated on the base that now holds it, after redoing the push-lock and lease waits; a half whose base moved before it landed is a CANT leaf. A red half splits again until the depth is spent; a leaf (a single PR, or the depth floor) keeps the old CANT and FLAKY meaning and lands nothing: it prints `CANT leaf #...`. Only a CANT leaf continues the split; an infra or policy failure (exit 2, 4, or 126 and up) stops it. The run always ends with `SPLIT done: landed #...; CANT (not landed) #...; gates=<n>`, where landed is what `land_train.sh` actually merged and every other PR (dropped culprits included) is not landed. It exits 0 only when everything landed, otherwise 3 (or the abort code). The flake-on-base CANT (`flake ... never fails on the base`) never splits, because re-running a possible intermittent regression in halves would only re-roll the dice. A split does add independent rolls of the same full gate, but only for a failure that already re-failed every solo re-run at the union (REAL), and each half still lands only on its own green gate. Cost: one full gate per half, so a split that cannot find a green half costs at most `2^depth` extra gates. Test: `scripts/test-train-split.sh`.

## A load-flaky required gate is not a confirmed red — bounded-rerun the same commit to reclassify it before concluding a regression
Expand Down
2 changes: 2 additions & 0 deletions .claude/skills/agentic-delivery/references/retrospective.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ applies to code, applied to incident causes: a recurring root cause with
nothing new added between occurrences means nothing was actually learned the
first time.

Every finding that is fixed leaves one pin test, and every train runs all pin tests. Before filing a new finding, run the fingerprint duplicate check (`scripts/file_finding.py`) so a recurrence reopens the old finding instead of adding a second one.

## Cross-reference — the reliability contract this feeds

`role-coverage.md`'s Platform/DevOps/SRE lens (`deep-code-review` skill) owns
Expand Down
8 changes: 5 additions & 3 deletions .claude/skills/agentic-delivery/scripts/train_flake.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,14 @@
# (FLAKE_HOST, default hostname -s) and invocation hash (sha256 of FLAKE_INVOCATION, default BROWSER_CMD) must match, else exit 2:
# a baseline from another machine or another command line makes failures look new (or hides new ones). Missing header = refused.
# Fresh store: FLAKE_SETUP_CMD (optional, via bash -c) runs before EVERY isolated re-run with TEST=<id>, FLAKE_RUN=<n> and
# FLAKE_STORE=<fresh empty dir from mktemp -d under $TMPDIR, left for OS temp cleanup, path printed to stderr>; it must seed that store, and RERUN_CMD sees the same FLAKE_STORE. A re-run that
# FLAKE_STORE=<fresh empty dir from mktemp -d under $TMPDIR, removed on exit, path printed to stderr>; it must seed that store, and RERUN_CMD sees the same FLAKE_STORE. A re-run that
# reuses the run's own store fakes real failures. Hook failure exits 2 (fails closed). Retries are per test title, never per file.
set -uo pipefail
inv=$(printf %s "${FLAKE_INVOCATION:-${BROWSER_CMD:-}}" | { shasum -a 256 2>/dev/null || sha256sum; } | cut -c1-16)
host=${FLAKE_HOST:-$(hostname -s)}
T=${TMPDIR:-/tmp}
# made: the mktemp paths this script created (stores, out file); the EXIT trap removes only those, skipping empty entries.
made=(); trap 'for x in ${made[@]+"${made[@]}"}; do [ -z "$x" ] || rm -rf -- "$x"; done' EXIT
# Ledger env (see the FLAKE_LEDGER_DIR note above): move it into shell variables and unset it so the code under test (RERUN_CMD, FLAKE_SETUP_CMD) never inherits the path.
led=${FLAKE_LEDGER_DIR:-} qstdin=${FLAKE_QUARANTINE_STDIN:-}; unset FLAKE_LEDGER_DIR FLAKE_QUARANTINE_STDIN FLAKE_QUARANTINE_JSON qjson
# norm: the ONE id normalizer (strip trailing whitespace incl. CR, nothing else). Every emit and match of a FAIL/REAL id goes through it
Expand All @@ -30,7 +32,7 @@ norm() { sed -E 's/[[:space:]]+$//'; }
[ "${1:-}" != --norm ] || { norm; exit $?; }
# one_run <id> <n>: fresh store (mktemp -d, FLAKE_STORE), optional FLAKE_SETUP_CMD hook, then RERUN_CMD alone. rc 125 = hook failed.
one_run() {
local d rc=0; d=$(mktemp -d "$T/flake-store.XXXXXX") || return 125; echo "train_flake: store $d (left for OS temp cleanup)" >&2
local d rc=0; d=$(mktemp -d "$T/flake-store.XXXXXX") || return 125; made+=("$d"); echo "train_flake: store $d (removed on exit)" >&2
export TEST=$1 FLAKE_RUN=$2 FLAKE_STORE=$d
if [ -z "${FLAKE_SETUP_CMD:-}" ] || bash -c "$FLAKE_SETUP_CMD" >/dev/null 2>&1 </dev/null; then python3 "$(dirname "$0")/stage_run.py" --name train-rerun -- bash -c "$RERUN_CMD" >"${RERUN_OUT:-/dev/null}" 2>&1 </dev/null || rc=$?
# stage_run exit 75 (heavy slot DEFER) or 124 (stage timeout) means the re-run never gave a verdict: infra (125), never a red/real failure.
Expand All @@ -44,7 +46,7 @@ if [ "${1:-}" = --one ]; then [ -n "${2:-}" ] && [ -n "${RERUN_CMD:-}" ] || { ec
# 1 = not pre-existing (RERUN_CMD must print `FAIL <id>` only for tests that actually ran), 125 = fresh store/hook failed or bad bounds (fail closed).
if [ "${1:-}" = --base ]; then
[ -n "${2:-}" ] && [ -n "${RERUN_CMD:-}" ] || { echo "usage: RERUN_CMD=... train_flake.sh --base <id>" >&2; exit 2; }
runs=${FLAKE_RUNS:-3} need=${FLAKE_REAL_AT:-$runs}; f=0 p=0; o=$(mktemp "$T/flake-out.XXXXXX") || exit 125
runs=${FLAKE_RUNS:-3} need=${FLAKE_REAL_AT:-$runs}; f=0 p=0; o=$(mktemp "$T/flake-out.XXXXXX") || exit 125; made+=("$o")
bid=$(printf %s "$2" | norm)
case $runs$need in ''|*[!0-9]*) exit 125 ;; esac; [ "$need" -ge 1 ] && [ "$need" -le "$runs" ] || exit 125
for n in $(seq "$runs"); do
Expand Down
4 changes: 2 additions & 2 deletions .claude/skills/deep-code-review/INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Generated by `scripts/skill_index.py` (do not hand-edit; rerun it after changing
| `references/appsec-ssti.md` | the target or diff compiles a server-side template from anything but a static file: render_template… | 507 |
| `references/appsec-supply.md` | the target or diff adds or bumps a dependency, touches a lockfile or an install-time script, edits… | 2623 |
| `references/appsec-tokens.md` | the target or diff issues or verifies JWTs, implements OAuth or OIDC, rotates refresh tokens, or mi… | 1749 |
| `references/appsec-verify.md` | you are about to report a security finding, when a project asks to tune which low-signal security c… | 1857 |
| `references/appsec-verify.md` | you are about to report a security finding, when a project asks to tune which low-signal security c… | 1965 |
| `references/billing-correctness.md` | the target meters usage, runs subscriptions, charges cards, handles invoices or tax, or processes r… | 2168 |
| `references/branch-and-merge-hygiene.md` | Read this on a FULL / repo-level review of a git repo with more than one branch, or whenever the re… | 7979 |
| `references/code-structure.md` | the review judges module boundaries or coupling: a FULL review of a codebase that has grown past on… | 1612 |
Expand Down Expand Up @@ -137,7 +137,7 @@ Generated by `scripts/skill_index.py` (do not hand-edit; rerun it after changing
| `references/testing-ui.md` | the target renders a UI or ships browser / E2E specs. | 4868 |
| `references/threat-modeling.md` | a change adds a trust boundary, an auth path, a data flow to a new party, an agent or tool capabili… | 2574 |
| `references/time-date-correctness.md` | reviewing anything that stores, computes, schedules, or renders timestamps, durations, recurring ev… | 4809 |
| `references/tool-hub.md` | a review should lean on external scanners via the tool hub (workflows, Python, Go, SBOM/lockfiles,… | 1127 |
| `references/tool-hub.md` | a review should lean on external scanners via the tool hub (workflows, Python, Go, SBOM/lockfiles,… | 1248 |
| `references/ux-components.md` | the target or diff renders one concept in more than one place: a shared component or design token,… | 7235 |
| `references/ux-dataviz.md` | the target or diff renders a chart, sparkline, heat-map or calendar grid, a KPI / metric delta, a c… | 2787 |
| `references/ux-gates.md` | a UI change is claimed verified on screenshot evidence, the project's auto-merge or CI policy gates… | 3756 |
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/wave-ak-train-flake-cleanup-rules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
### Fixed
- agentic-delivery: `train_flake.sh` now removes the flake-store and flake-out temp dirs it creates (EXIT trap over only its own mktemp paths) instead of leaving them for OS cleanup. `scripts/test-train-flake-cleanup.sh` proves none remain.
### Added
- agentic-delivery: short rules in `red-gate-triage.md` (rerun failing titles solo with one worker, 2 of 2 passes is a flake, paired base run before a bisect, AIMD batch window), `retrospective.md` (one pin test per finding in every train, fingerprint duplicate check before filing) and `pipeline-optimizer.md` (each pipeline fix removes a knob or code path).
size-budget-raise: .claude/skills/agentic-delivery/references/red-gate-triage.md 27186→27616 one short rule paragraph added: solo rerun, paired base run, AIMD window
size-budget-raise: .claude/skills/agentic-delivery/references/retrospective.md 4844→5095 one short rule added: pin test per finding, fingerprint dedupe
size-budget-raise: .claude/skills/agentic-delivery/references/pipeline-optimizer.md 19352→19480 one short rule added: deletion budget
1 change: 0 additions & 1 deletion scripts/mktemp-trap-baseline.tsv
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Shipped shell scripts that call mktemp without a trap EXIT; frozen, shrink only (bg_admit.py untrapped).
.claude/skills/agentic-delivery/scripts/train_flake.sh
scripts/test-auto-context-reset.sh
scripts/test-conflict-free-prs.sh
scripts/test-cost-statusline.sh
Expand Down
6 changes: 3 additions & 3 deletions scripts/size-budgets.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
.claude/skills/agentic-delivery/references/operating-discipline.md 11608
.claude/skills/agentic-delivery/references/production-grade-product-playbook.md 6635
.claude/skills/agentic-delivery/references/project-state.md 7540
.claude/skills/agentic-delivery/references/retrospective.md 4844
.claude/skills/agentic-delivery/references/retrospective.md 5095
.claude/skills/agentic-delivery/references/roles.md 27320
.claude/skills/agentic-delivery/references/support-ops.md 4455
.claude/skills/agentic-delivery/references/template-adr.md 2641
Expand Down Expand Up @@ -198,7 +198,7 @@ README.md 16118
.claude/skills/deep-code-review/references/compact-mode.md 2491
.claude/skills/deep-code-review/references/full-audit.md 6584
.claude/skills/deep-code-review/references/nextjs-app-router.md 2151
.claude/skills/agentic-delivery/references/red-gate-triage.md 27186
.claude/skills/agentic-delivery/references/red-gate-triage.md 27616
.claude/skills/agentic-delivery/references/worktree-isolation.md 16482
.claude/skills/communication-structure/references/leadership-status-update.md 2003
.claude/skills/idea-critic/references/anti-rationalization.md 1330
Expand All @@ -213,7 +213,7 @@ README.md 16118
.claude/skills/agentic-delivery/INDEX-scripts.md 15782
.claude/skills/deep-code-review/INDEX.md 20406
.claude/skills/deep-code-review/INDEX-scripts.md 5356
.claude/skills/agentic-delivery/references/pipeline-optimizer.md 19352
.claude/skills/agentic-delivery/references/pipeline-optimizer.md 19480
.claude/skills/agentic-delivery/references/focus-kit.md 5235
.claude/skills/communication-structure/references/artifact-templates.md 7029
.claude/skills/deep-code-review/references/code-structure.md 6449
Expand Down
7 changes: 7 additions & 0 deletions scripts/test-ci-gates.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4056,6 +4056,13 @@ else
record 1 "train flake-on-base: a FLAKE lands only if the test also fails on the base, else CANT"; tail -8 "$WORK/train-flake-base.log"
fi

# train flake cleanup: own per-feature file (scripts/test-train-flake-cleanup.sh), one case here.
if bash "$ROOT/scripts/test-train-flake-cleanup.sh" >"$WORK/train-flake-cleanup.log" 2>&1; then
record 0 "train flake cleanup: temp dirs removed on exit, triage rules present"
else
record 1 "train flake cleanup: temp dirs removed on exit, triage rules present"; tail -8 "$WORK/train-flake-cleanup.log"
fi

# train split on CANT: own per-feature file (scripts/test-train-split.sh), one case here.
if bash "$ROOT/scripts/test-train-split.sh" >"$WORK/train-split.log" 2>&1; then
record 0 "train split: an unattributed red train re-gates halves, lands only green halves, leaves keep CANT"
Expand Down
3 changes: 2 additions & 1 deletion scripts/test-fleet-lessons.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ FLAKE_SETUP_CMD='echo seed >"$FLAKE_STORE/s"' RERUN_CMD='[ -n "${FLAKE_STORE:-}"
ok $? "bisect helper: --one gives a fresh seeded store (a stale shared store would fail)"
RERUN_CMD='[ -n "${FLAKE_STORE:-}" ]' bash -c 'unset FLAKE_STORE; bash -c "$RERUN_CMD"'; [ $? -ne 0 ]; ok $? "without the helper FLAKE_STORE is unset, so a stale-store re-run fails"
grep -q 'train_flake.sh" "\$1"' "$SC/train_land.sh"; ok $? "train_land bisect re-runs through train_flake.sh triage (fresh store per run)"
! grep -nE '\brm\b' "$SC/train_flake.sh"; ok $? "train_flake.sh contains no rm"
rmlines=$(grep -cE '\brm\b' "$SC/train_flake.sh"); trapok=$(grep -cF "do [ -z \"\$x\" ] || rm -rf -- \"\$x\"; done' EXIT" "$SC/train_flake.sh")
[ "$rmlines" -eq 1 ] && [ "$trapok" -eq 1 ]; ok $? "train_flake.sh: only rm is the EXIT trap over its own mktemp paths"

# --- timebox kills the whole process group (grandchild included), only its own ---
python3 -I - "$SC" "$WORK/gc.pid" <<'PY'
Expand Down
27 changes: 27 additions & 0 deletions scripts/test-train-flake-cleanup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# train_flake.sh removes the flake-store/flake-out temp dirs it makes (EXIT trap), on triage and --base paths, and leaves
# unrelated TMPDIR entries alone. Also pins the triage-rule wording in red-gate-triage.md. Stubs only; writes under $WORK.
set -uo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
SC="$ROOT/.claude/skills/agentic-delivery/scripts"
WORK="$(mktemp -d "${TMPDIR:-/tmp}/dcr-test-train-flake-cleanup.XXXXXX")"
trap '[ -n "$WORK" ] && rm -r "$WORK" 2>/dev/null' EXIT
unset GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL
pass=0 fail=0
ok() { if [ "$1" -eq 0 ]; then echo "PASS $2"; pass=$((pass + 1)); else echo "FAIL $2"; fail=$((fail + 1)); fi; }

mkdir -p "$WORK/tmp"; echo keep >"$WORK/tmp/unrelated"; echo x >"$WORK/id"
export SEEN="$WORK/seen" FLAKE_INVOCATION=x RERUN_CMD='[ -d "$FLAKE_STORE" ] && : >"$SEEN" && echo "FAIL x"; false'
TMPDIR="$WORK/tmp" bash "$SC/train_flake.sh" "$WORK/id" >/dev/null 2>&1
[ -f "$WORK/seen" ]; ok $? "positive control: triage run saw its flake store"
left=$(find "$WORK/tmp" -mindepth 1 -name 'flake-*' | wc -l | tr -d ' ')
[ "$left" -eq 0 ]; ok $? "triage path leaves no flake-store/flake-out dirs ($left left)"
TMPDIR="$WORK/tmp" bash "$SC/train_flake.sh" --base x >/dev/null 2>&1
left=$(find "$WORK/tmp" -mindepth 1 -name 'flake-*' | wc -l | tr -d ' ')
[ "$left" -eq 0 ]; ok $? "--base path leaves no flake-store/flake-out dirs ($left left)"
[ -f "$WORK/tmp/unrelated" ]; ok $? "unrelated TMPDIR entry untouched"
grep -q "solo with one worker" "$ROOT/.claude/skills/agentic-delivery/references/red-gate-triage.md" \
&& grep -q "AIMD" "$ROOT/.claude/skills/agentic-delivery/references/red-gate-triage.md"; ok $? "red-gate-triage.md carries solo-rerun and AIMD rules"
grep -q "Deletion budget" "$ROOT/.claude/skills/agentic-delivery/references/pipeline-optimizer.md"; ok $? "pipeline-optimizer.md carries deletion budget"
grep -q "pin test" "$ROOT/.claude/skills/agentic-delivery/references/retrospective.md"; ok $? "retrospective.md carries pin-test rule"
echo "$pass passed, $fail failed"; [ "$fail" -eq 0 ]
Loading