Detect column misalignment by content, not by missing fields - #18
Merged
Conversation
The first version keyed off rows carrying too few fields. Writing the CSV back out pads every row to the full column count, so one --fix pass erased the signal while leaving the defect in place: the CI run after the previous merge reported "562/562 rows carry all columns; 0 misaligned" on a dataset where 191 rows still serve a privacy policy or a community forum as their documentation URL. misalignedAs() now runs over every row and keys off the URL itself, so the finding survives normalisation. Per-column statistics are gated on rows that are both complete and unshifted, which is the honest denominator: 371 of 562. Tests cover the detector, with a note on why it is content-based, so this cannot regress silently again. Also bumps the workflow to Node 22 — runners now default to Node 24 and warn on 20. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #17, fixing a regression that the first scheduled run exposed.
The bug
The alignment detector keyed off rows carrying too few fields. But writing the CSV back out pads every row to the full column count — so one
--fixpass erased the signal while leaving the defect completely in place.The dispatch run after #17 merged duly reported:
on a dataset where 191 rows still serve a privacy policy, community forum or rate-limit page as their
Official_Documentation_URL.Microsoft Graph People APIstill hands youprivacy.microsoft.com. The detector had gone quiet about a defect it was built to find — the worst failure mode available to a monitor.The fix
misalignedAs()now runs over every row and keys off the URL content, so the finding survives normalisation. Per-column statistics are gated on rows that are both complete and unshifted — 371 of 562, which is the honest denominator.Tests cover the detector directly, with a comment recording why it is content-based, so it cannot regress silently again.
npm test→ 16 passing.What the run proves
Re-probing after #17's 1,472 corrections landed:
okmoveddeadThe self-healing pass does what it claims: redirect drift went from the dominant failure mode to noise. Residual
movedis redirect chains that resolve one hop further on the second pass./llms.txtadoption also moved: 79 of 167 hosts (47%), up from 74/164 measured a few hours earlier — some of that is the larger host set, and this is exactly why the number is tracked per run rather than quoted once.Also
Still outstanding for the maintainer
The weekly PR step fails with
GitHub Actions is not permitted to create or approve pull requestsuntil Settings → Actions → General → "Allow GitHub Actions to create and approve pull requests" is enabled. Every other step succeeds and the full report uploads as an artifact regardless.🤖 Generated with Claude Code