Purge fabricated case studies from the blog archive, highest-traffic first - #583
Merged
Conversation
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
pftg
force-pushed
the
purge-fabricated-case-studies
branch
from
August 22, 2026 11:53
b0b3357 to
5091f8f
Compare
Executes the standing canon policy (claims-canon.md, 2026-08-20): "extend the
purge to ranking legacy posts, highest-impression first."
Priority came from a live GSC pull (90d, 2026-05-24 → 2026-08-20, page
dimension filtered to /blog/), NOT from how bad each claim looked. That
ordering overturned my own starting assumption - see below.
Removed, all "invented client work" in the canon's sense (an anonymous company,
precise metrics, no client behind it and no possible source):
- `rails-8-solid-cache-performance-redis-migration` (**4,891 impr, pos 9.6** -
the highest-traffic carrier): two case studies inventing a content platform
($450→$125/month, 72%, 85% hit rate, +12ms) and a retailer ($320/month).
Internal link preserved.
- `laravel-performance-monitoring-complete-apm-comparison-guide` (**1,729 impr**):
a 265-line "Real-World Performance Optimization Case Studies" section with two
invented case studies carrying fabricated APM readings (8734ms, 94% DB time,
12456 queries). Its closing line - "the pattern we see repeatedly" - is the
recurrence-generalisation shape that is itself a banned de-fabrication hatch.
- `rails-event-structured-logging-8-1` (**594 impr, pos 10.5**): the TL;DR
claimed "we migrated four production apps and the false-positive alert rate
dropped by 60%". The string occurs exactly once in the repo - no source.
Replaced with the mechanism claim, which is a property of Rails 8.1 and needs
no engagement behind it.
Two method notes:
1. **Impressions, not indignation.** I had flagged
`how-to-manage-developers-when-you-cant-code` ("200+ times with clients") as
the priority because it is `featured: true` and ICP-facing. It has **4
impressions in 90 days**. `featured` is a site-internal flag, not traffic.
The canon's own rule caught my error: a fabricated story on a page nobody
reads is a liability, on a ranking page it is what a prospect sees first.
2. **The 500-row pull hit its cap (`has_more: true`)** - Trap A. Harmless here
only because rank 500 has 1 impression, so anything absent is zero-traffic.
Stated rather than assumed.
Tradeoff worth naming: the laravel deletion removed working Laravel optimisation
code along with the invented framing. The code was persuasive BECAUSE of the
fabricated results around it; salvaging it is a separate call.
Gate: `bin/hugo-build` green. Content-only diff (markdown prose), so the visual
suites correctly do not apply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
Follow-up to the previous commit, after Paul confirmed all three first-person
claims I had flagged were invented rather than real engagements. That verdict
made the whole class suspect, so I re-swept - and the re-sweep found the first
pass had been looking for the wrong thing.
**The candidate regex was the defect.** It keyed on "N clients/companies/times",
which misses every fabrication phrased as a case study or as "in our
experience". Searching the STRUCTURE instead (`^#{2,4} .*Case Stud`,
`Real-World Results`) returned 33 files, and the two biggest carriers were
absent from the original list entirely:
- `langgraph-workflows-state-machines-ai-agents` - **40,025 impressions**, 8x
anything in the first pass. Carried four third-party claims (Uber, LinkedIn,
Klarna "80% reduction", AppFolio "2x accuracy") each tagged "(figures
unverified)". That tag is a half-measure: the number still does the
persuading and the disclaimer is what a reader skips. Also removed a
"Download our free Workflow Patterns Library" CTA - there is no download, the
templates are listed inline.
- `propshaft-vs-sprockets` - **6,194 impressions, pos 10.4**. A 244-line
invented case study, plus "in our experience" timing figures (45-60s → under
5s) and an "Our typical results" benchmark table with no measurement behind
it. Replaced with the command to measure their own app, which is the number
that actually decides their migration.
Also cleared: `solid-queue-vs-sidekiq` (50,000+ jobs daily / 35% cost
reduction), `rails-8-docker-deployment` (a "B2B SaaS platform with 50,000
active users"), `cost-optimization-llm-applications` (a chatbot with a
$3,400/month saving), the AI-integration guide (200+ clients, 3 clients),
pgvector (15+ teams), crewai (a duplicated unsourced cost claim), and the three
first-person claims Paul confirmed invented.
Three recurrence-generalisation openers went too ("the pattern we see most
often", "the pattern across the rescues we've taken"). That shape is the
default escape hatch when a fabricated specific is removed, and it is banned
for exactly that reason - it keeps the authority of experience while shedding
the falsifiable part.
**Salvage (Paul approved):** the Laravel N+1 and memory-exhaustion fixes are
restored as "Two Fixes APM Points You Straight At" - the before/after code kept,
the invented companies and measurements dropped, and both now tell the reader to
measure their own numbers.
Gate: `bin/hugo-build` green. Content-only diff, so the visual suites do not
apply.
NOT covered, named so it is not mistaken for done: 13 dev.to-sourced posts carry
claim-shaped strings (two at 3,285 and 2,934 impressions); the ratchet skips
them as the original authors' stats, which is a TEST-scoping decision and not
editorial absolution. And `marketing_copy_test.rb` still excludes
`content/blog/**`, so nothing prevents regression here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
…anual Three hand-sweeps in one session each found carriers the previous one missed, because each keyed on the wrong surface: a "N clients" regex missed everything phrased as a case study, a case-study-heading sweep missed everything phrased as "in our experience", and both missed frontmatter. That is claims-canon.md's own finding reproducing itself - "these are ratcheted mechanically, not by review; manual sweeps under-count badly". BANNED cannot catch this class, because a fabricated case study is built from ordinary words. What it has instead is a SHAPE, and shape is greppable: - a case-study heading in a technical post (real client work lives in content/clients with a named client) - "in our experience" / "the pattern we see" - the recurrence-generalisation hatch, what a fabricated specific collapses into when the number is dropped but the authority is kept - "(figures unverified)" - a number tagged rather than removed **Baseline 17, and proven exact.** Set to the measured count, then dropped to 16 to confirm the gate actually fires rather than sitting on slack - it failed with "Expected 17 to be <= 16", then restored. That step is not ceremony: the rendered baseline in this same file sat at 14 against an actual 11, and the three spare hits swallowed a planted phrase whole. 8 of the 17 survivors are ONE deferred decision, not 8 defects: the fractional-CTO posts already fall under Paul's 2026-08-21 ban on fractional-CTO title claims and need a wholesale call (rewrite, redirect or retire). Editing their case-study headings first would bury that decision under a cosmetic fix. Also corrected a comment that had become false and would mislead the next reader: SURFACES says `content/blog/**` is excluded, which is true of THAT pass only - the rendered pass already covers `blog/**/*.html`, and now the fabrication ratchet covers blog source. I repeated that stale reading myself earlier today before checking. It now says so explicitly. Content fixes in the same commit, both found by the frontmatter sweep the markers now make unnecessary: - `pgvector-rails-tutorial` carried an invented anecdote as its OPENING - a named-in-all-but-name CTO, fabricated benchmarks (89ms vs 52ms), "his annual savings: $7,200" - plus "Save $6,000/year vs Pinecone" in the frontmatter description. Frontmatter is published copy; this is the exact defect .okf/log.md records for twitter_description. - `cost-optimization-llm-applications` claimed "30-60% in our experience" in the description, the intro, a summary bullet and the conclusion. Gates: `bundle exec ruby -Itest test/unit/marketing_copy_test.rb` - 4 runs, 10 assertions, 0 failures. `bin/hugo-build` green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
Paul ran /blog-next repeatedly and never got a post, and concluded the pipeline was broken. It was not broken - it was half-built, and the missing half had no way in. /blog-next never writes, by design: Paul asked for the split. But the delivery half existed only as AGENTS (blog-post-coordinator, blog-batch-orchestrator), which a user cannot invoke by typing. So there was literally no command that produced a post. The split was right; shipping only one side of it was not. Two skills close it: **`blog-write`** - the delivery half, invokable. Owns blog-pipeline.md STEP 4 onward and deliberately does not restate it, so a gate correction lands in one place. Handles UPGRADE as a first-class mode, not an afterthought - the one approved item in the queue right now IS an upgrade, so a writer that only created new posts would still have been unable to do the only queued job. Carries three exits, and explicitly does NOT have HOLD: by the time work reaches it the decision to write has been audited, so "produce nothing" is a falsified premise to hand back, not a quiet non-result. **`blog-operator`** - one door that decides which hand to use, then keeps going (WIP=1, re-decide after each unit). Its ordering is the opinionated part: REPAIR > UPGRADE > WRITE > RESTOCK. Credibility defects outrank new content because a founder who catches one invented claim stops believing the rest of the page, and no new post recovers that. An upgrade to a page that already ranks beats a new post starting from zero. Both state plainly that volume is not the goal - a tenth mediocre post dilutes the nine and gives a sceptical reader more surface to find a flaw. The .gitignore trap fired again, exactly as it did for blog-next: both skills were invisible to `git status` until named. `.claude/skills/*` is ignored as claude-flow scaffolding, so every hand-authored skill needs its own negation. Checked with `git status --untracked-files=all` rather than `git check-ignore`, which returns exit 0 either way and reads as confirmation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
/codex:review returned four findings against this branch. All four reproduced against the tree, so all four are fixed here. **False positive: a real case study counted as fabrication.** `async-remote-xp-practices` writes up THIS repository's CSS migration - the commits are in this git history, which is as verifiable as a case study gets - and the heading marker classed it with the invented ones. A case-study heading is a SUSPICION, not a verdict: the shape is identical either way. Verified subjects are now allowlisted by path, and the entry test is whether the write-up names a subject a reader could go and check. "A medium-sized content platform" is not a subject; that is the whole difference. Baseline 17 -> 16, because one of the 17 was never a defect. **False negative: ordinary Markdown wrapping walked straight through the gate.** `/\bin our experience\b/` returns false against `"in our\nexperience"` - verified directly, not assumed. A line-by-line scan therefore has a hole that any re-wrap opens. Prose markers now match the whitespace-collapsed document, which is exactly why the rendered pass in this same file collapses before matching. Heading markers stay line-based: Markdown ends a heading at the newline, and a line number is what you want when fixing one. **A confident replacement is not a fix.** My propshaft rewrite removed an unsourced timing figure and asserted that precompilation "stops being a build step that scales with your asset count". Propshaft still walks, fingerprints and copies every asset, so it does scale - what drops is the per-asset cost. Trading a fabricated number for an inaccurate mechanism is the worse outcome, because the mechanism reads as reasoning rather than as a claim to check. Also converted two em dashes in the same sentence to hyphens per the voice rule. **The OKF entry the ENFORCED rule requires.** A durable gate landed with its rationale only in test comments. Added `.okf/content/fabrication-ratchet.md` plus its index line and a dated log entry, in this commit rather than a later one. Also removed the baseline comment's reasoning about the fractional-CTO posts - Paul took those off the table entirely, and a comment that argues about them is the opposite of ignoring them. Gates: marketing_copy_test 4 runs / 10 assertions / 0 failures. Ratchet proven exact again at the new count - dropped to 15, failed with "Expected 16 to be <= 15", restored. `bin/hugo-build` green. `okf_validate --strict`: conformant, zero errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
pftg
force-pushed
the
purge-fabricated-case-studies
branch
from
August 22, 2026 12:16
d1e2243 to
f46999f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #580 (docs-only; merge that first).
Executes the standing canon policy from
claims-canon.md(2026-08-20) — "extend the purge to ranking legacy posts, highest-impression first" — rather than a fresh alarm. The policy already existed; it had just never been run.Priority came from GSC, and it overturned my starting assumption
I began by flagging
how-to-manage-developers-when-you-cant-code("200+ times with clients") because it isfeatured: trueand ICP-facing. A live pull (90d, page dimension,/blog/filter) shows it has 4 impressions in 90 days.featuredis a site-internal flag, not traffic. I had sorted by how bad the claim looked; the canon's own rule says sort by who sees it.The candidate regex was the real defect
The first pass searched for
N clients/companies/times. That misses every fabrication phrased as a case study or "in our experience." Searching the structure instead (^#{2,4} .*Case Stud,Real-World Results) returned 33 files — and the two largest carriers were absent from the first list entirely:langgraph-workflows-state-machines-ai-agentspropshaft-vs-sprocketsrails-8-solid-cache-...-redis-migrationsolid-queue-vs-sidekiqlaravel-performance-monitoring-...rails-8-docker-deploymentcost-optimization-llm-applicationsrails-event-structured-logging-8-1"(figures unverified)" is a half-measure. The langgraph post tagged its Klarna and AppFolio numbers rather than removing them. The number still does the persuading; the disclaimer is what a reader skips. Same shape as the frontmatter case in
.okf/log.md— a softening that reaches the body but not the metatag still ships the claim.Recurrence-generalisations went too
Three openers — "the pattern we see most often", "the pattern across the rescues we've taken" — are the default escape hatch when a fabricated specific is removed. They keep the authority of experience while shedding the falsifiable part, which is exactly why the shape is banned.
Salvage
The Laravel deletion took working optimisation code with it. Restored (approved) as "Two Fixes APM Points You Straight At": before/after N+1 and memory-exhaustion code kept, invented companies and measurements dropped, and both now tell the reader to measure their own numbers. Same move on propshaft — the invented benchmark table became the command to time their own app, which is the number that actually decides their migration.
Gates
bin/hugo-build— green.Not covered — named so it is not mistaken for done
marketing_copy_test.rbstill excludescontent/blog/**. Nothing prevents regression here. Per the log entry on the fractional-CTO rule, the honest fix is a scoping decision, not a regex.🤖 Generated with Claude Code
https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg