feat(playability): 300-game human-vs-AI study + per-level reports - #15
Conversation
Headless harness (scripts/playability/): a fixed measured-reactive human policy (player1) vs a fixed 4-AI cast, so difficulty is the only variable. Plays 100 games/level, emits per-game JSON records + narratives, and a deterministic aggregator so the reports cite exact numbers. docs/playability/<level>/: games.jsonl, games.md, play-summary.md, trend-analysis.md, playability-outcome.md (3 playability outcome reports). Headline: normal is the most playable (83% survivor); easy 58%; hard is degenerate — 100/100 pyrrhic, human survival 0%, near-identical script every game. Human win% rises with difficulty (18/28/37) while survival falls (8/18/0) because hard 'wins' are all pyrrhic. Also surfaced a reporting artifact: pyrrhic winner = last-to-die, so win%/placement mislead unless read against survival%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
🟢 Claude BugBot Analysis
Reviewed the four new TypeScript files (aggregate.ts, humanPolicy.ts, run.ts, runner.ts); verified Order/ResolutionEvent/Leader/Stockpile shapes and the validateOrderSequence/totalApCost/planAi/isHuman signatures against src/engine, and all usages match. No genuine bugs found — the rest of the diff is auto-generated playability report data (docs/playability/**/*.jsonl and *.md).
No bugs were detected in this PR.
…ence Adds extra.ts (deterministic replay instrumenting AI warhead yields, fired-vs- built ratio, and launch cadence by game-third) and a findings report answering: - warhead mix: small-dominant everywhere, rising with difficulty (easy 71% → normal 89% → hard 98.5% small; hard never fired a large warhead in 100 games); - hoarding: none — AI fires 92-96% of what it builds, front-loaded (~75-80% of launches in the first two game-thirds); - game length: easy 15.2 / normal 17.5 / hard 17.4 avg rounds; - explains the easy<normal survival paradox: easy's 30% randomness manifests as 3x more large-warhead strikes and shorter, more-pyrrhic games, which punish a cautious human more than legible normal AI does. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔴 Claude BugBot Analysis
Found 1 potential bug in this PR.
low: 1
The diff is mostly generated docs/data; the actual code changes (scripts/playability/*.ts) are clean and type-consistent with the engine APIs they call. Only one minor correctness issue was found: aggregate.ts's avgEliminatedRound silently defaults to 0 (instead of null/N-A) when no games in a batch have a human elimination, which can produce misleading statistics.
Adds balanced + turtle policies and a policy-selectable runner; runs each of cautious / aggressive / balanced as a lone human vs the 4-AI cast, 100 games per difficulty (900 games), with a strategy-comparison report. Findings: balanced >= cautious >> aggressive, but the human's ceiling is low (<=7% win vs 4 AI). All-in dies ~2x faster (elim round ~7 vs ~13-16) and never survives — opening fire spreads grudges and draws concentrated retaliation. Balanced is the only strategy to score on hard (5% win, best placement). Escalating 'eliminate the attacker' retaliation lowered survival vs simple measured retaliation — proportionate response is self-defeating here. Also fixes the BugBot LOW finding: aggregate.ts avgEliminatedRound now returns null (not 0) when no human elimination occurs, so it can't misread as 'round 0'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Claude BugBot Analysis
Reviewed the added scripts/playability/*.ts files (the only non-generated code in this diff; the rest is generated markdown/jsonl playability data). Cross-checked all Order/ResolutionEvent field usages, imports, and AP/stockpile validation logic against src/engine/types.ts and src/engine/orders.ts — all signatures, field names, and sequencing logic are correct. No genuine bugs found.
No bugs were detected in this PR.
1 participant
What
A headless playability study: 300 games (100 per difficulty) with a human player (player1, a fixed "measured-reactive" policy) against a fixed 4-AI cast, so difficulty is the only independent variable. Per-game records + narratives, a deterministic aggregator, and per-level trend + playability reports — produced by an agent team (3 play agents, 3 collators).
Deliverables
scripts/playability/—runner.ts(game loop + per-game record/narrative),humanPolicy.ts(the human proxy),aggregate.ts(exact stats),run.ts(CLI). Run:npx vite-node scripts/playability/run.ts <level> 100 docs/playability.docs/playability/<level>/for easy/normal/hard —games.jsonl(100 records),games.md(narratives),play-summary.md,trend-analysis.md, andplayability-outcome.md(the 3 playability outcome reports).Headline findings (all from the aggregator)
winRate/placementread like dominance whilesurvivalRateis 0 — they mislead unless read together.Top recommendations (detailed per level in the reports): make a win-alive state reachable on hard (throttle AI launch volume / buff defensive economy), add variance so hard isn't two scripted games, and fix pyrrhic scoring so the scoreboard stops crediting a corpse as 1st.
Notes
tsc --noEmitclean); every cited number comes fromaggregate.ts, not hand-counting.🤖 Generated with Claude Code