Stop the cleaner corrupting rows, and repair the rows already shifted

[in-c0]

The dataset's worst defect was never link rot. utils/verify_and_clean_csv.py dropped a URL it judged broken instead of blanking it:

for url in urls:
    if is_broken_url(url):
        print(f"URL for {api_name} is invalid: {url}")
    else:
        cleaned_row.append(url)   # broken URLs simply vanish

Dropping shortens the row, so every later value slides one column left and is filed under the wrong heading. Microsoft Graph People API served its privacy statement as its documentation URL. Those rows were not stale — they were mislabelled, which is worse, because the consumer gets a confident wrong answer with nothing to indicate a problem.

It compounded, too: is_broken_url returned True on any request exception with a five-second timeout. A slow host, a TLS hiccup or a 403 from a bot-detecting CDN permanently deleted a good URL and shifted everything after it. The freshness checker measures 26–82 URLs as blocked on a given run depending on egress IP; every one of those would have been destroyed.

The fix

The cause — broken URLs are blanked in place, rows are padded to full width, and only a confirmed 404/410 counts as broken. (invalid_urls is also actually incremented now; it was always reported as zero.)

The existing damage — tools/realign-columns.mjs. Dropping preserves order, so the surviving URLs are still in their original relative sequence, just compressed leftward. Repair is therefore an order-preserving assignment of the observed URLs back onto the column slots, solved exactly with dynamic programming and scored on what each URL looks like. A URL that contradicts every still-free column is left blank and recorded in datasets/realignment.json rather than filed somewhere wrong — the same principle that motivates the whole change.

1,089 URLs moved across 369 rows. Five were blanked as unplaceable.

Two of my own bugs, corrected

• Bare slack and discord in the community hint matched the vendors' own documentation hosts (docs.slack.dev), and tos matched inside "photos". This inflated the misalignment count I reported yesterday: the real figure was 148, not 191.
• Hints are now discounted against the API's own name, so AWS Security Hub API is not read as a misfiled security policy. Without that, any API named after a policy word looks permanently broken and no repair could reach zero.

Verification

tools/verify-alignment.mjs runs a negative control before reporting anything — it deliberately re-breaks a copy of the data the way the old cleaner did, and requires the detector to catch it. A detector that always returns zero also reports zero, so a clean result is worthless until the control has fired.

negative control (data deliberately re-shifted): 147 rows detected
api-docs-urls.csv:                               0 rows detected

npm test → 27 passing, including order-preservation and the name-aware hint cases.

Effect on the data

All 562 rows are now usable for per-column statistics, up from 371. Columns are properly populated for the first time — Privacy Policy 369 → 530 URLs, Developer Community 174 → 501, Security Policy 233 → 434.

The documentation column went the other way, 562 → 415 populated, and that is the point: 147 of those cells held a privacy policy or a forum link. The dataset now admits what it does not know instead of answering wrongly.

Overall health after the repair and a fresh probe: 93.5% ok, 0.6% moved, 3.5% dead. Release Notes is now the rot leader at 16.5% dead.

🤖 Generated with Claude Code