Evidence refresh: METR 2026 follow-up supersedes the 19% headline; DORA 2025 replaces 2024 as the delivery baseline

[richardsongunde]

Why this matters

RStack's research substrate (research/) and paper claims cite the METR early-2025 RCT and DORA 2024. Both have been superseded, and citing the old versions unqualified is now a claims-discipline violation of our own register (research/productivity-claims.md).

Verified in the #79 July-2026 research refresh (25/25 claims confirmed against primary sources):

1. METR follow-up (Feb 24, 2026) — https://metr.org/blog/2026-02-24-uplift-update/ — with newer agentic tools (Claude Code, Codex era), the measured effect is statistically indistinguishable from zero: returning developers −18% (CI −38%…+9%), newly recruited −4% (CI −15%…+9%). METR itself flags the data as an unreliable signal (selection bias) and is redesigning the experiment. The original page now carries an out-of-date banner. The durable finding is the perception gap (developers believed +20% while measured slower), not the 19% number.
2. DORA 2025 (Sept 2025, ~5,000 respondents) — https://dora.dev/dora-report-2025/ — AI adoption now has a positive relationship with throughput (a reversal of 2024) but still negative with delivery stability; 90% use AI, >80% self-report productivity gains. This is the strongest current empirical support for RStack's thesis: throughput-without-stability is exactly the failure mode a governed loop targets. Correlational, not causal.

Scope

• research/bibliography.md: update the METR entry (add the 2026 follow-up, scope the 19% figure to early-2025 tools, note METR's own reliability warnings); add DORA 2025 alongside/above 2024.
• research/productivity-claims.md: refresh the METR external-evidence row; add a DORA-2025 row (throughput/stability split); add an explicit "never cite 19% unqualified — pair the 2025 RCT with the 2026 update" rule; add the perception-gap claim as the durable citation.
• research/methodology.md: update the reference list and the allowed-wording example that leans on METR.
• research/paper-outline.md: "METR productivity caution" line updated to the perception-gap + stability framing.
• Paper master (separate repo, sdlc-rstack-paper): same corrections in paper.md/paper.tex/references.bib.

Guardrails

• Do NOT overclaim the null: METR characterizes both estimates as likely lower bounds; "no measured effect" must not become "AI does not help."
• DORA findings are survey correlations — keep causal language out.

Source

#79 epic research refresh comment: #79 (comment)

[richard-devbot]

In-repo items (1–4) shipped in PR #359 (Claude Main, solo). Refreshed bibliography.md, productivity-claims.md, methodology.md, paper-outline.md: METR now pairs the early-2025 RCT with the Feb-2026 follow-up (effect indistinguishable from zero) and elevates the perception gap as the durable finding; DORA 2025 is the delivery baseline (throughput up, stability still down, correlational); added a binding "never cite 19% unqualified / don't overclaim the null / DORA = correlation not causation" rule. Guardrails honored.

Item 5 (paper master, sdlc-rstack-paper) is NOT done — that repo is separate and not accessible from the SDLC-rstack working tree (gh repo view richard-devbot/sdlc-rstack-paper does not resolve). The same corrections (paper.md / paper.tex / references.bib) still need to be applied there. Leaving this issue's item-5 checkbox open until that lands, so it isn't falsely closed.

[richard-devbot]

Reopened: PR #359 landed in-repo items 1–4 on main, but item 5 (the separate sdlc-rstack-paper master repo) is not yet done — that repo isn't accessible from the SDLC-rstack tree. Keeping this open to track only item 5 (apply the same METR-2026 / DORA-2025 corrections to paper.md / paper.tex / references.bib there). The GitHub auto-close linkage fired on merge despite the edited body; reopened intentionally.