Repository navigation
fix(core): false-positive fixes from two site scans - #109
Merged
Merged
Conversation
The sitemap walk caps entries at 500 and child sitemaps at 10. A page missing from the part it read may be listed in the part it skipped, so discovery-index-coverage now returns notApplicable instead of failing. Found on a large retail site whose 2455-URL product sitemap filled the cap before the one-URL sitemap that lists the homepage.
Nested Review, Comment, Answer and Question dates record when someone else wrote, not when the page changed. Reading them as page signals reported product pages that publish no page date as divergent instead of unverifiable. Count lastmods within one hour as one build stamp. A generator that writes the clock per URL spreads one run over seconds, and exact string matching missed it.
The 14 per-bot robots.txt audits warned at 0.5 when a bot was allowed only through User-agent: *. Under RFC 9309 section 2.2.1 that is the same access a named group grants, and no vendor documents a difference. They now pass and name the group that applied. A blocked bot still fails. A missing or unreadable robots.txt is not applicable. Control-token copy no longer says Google-Extended or Applebot-Extended crawl.
fix(core): pass crawler bots allowed through the catch-all group
The audit warned when YouBot or AI2Bot was allowed through User-agent: *. Its only reason was that the policy could flip under a later blanket block; no source documents a difference, and RFC 9309 section 2.2.1 grants the same access. The dossier is corrected first. Inherited access now passes and names the rule that applied; a block still fails. A missing or soft-404 robots.txt is not applicable.
…allow fix(core): pass ai-bot-directives when bots inherit the catch-all
The audit reports contradictions between channels; a site with no signal in any channel has nothing to contradict. The dossier called silence a real cost with no source, while its own sources read absence as unknown or no preference. Dossier corrected first; notes stay in details.
fix(core): treat AI-usage silence as not applicable in coherence audit
When nothing advertises an RSL licence, the audit guesses /license.xml and /rsl.xml. A site that redirects both to its homepage got four conformance errors about the homepage HTML. A guessed path is now kept only when its body has an <rsl> element; otherwise the audit returns notApplicable.
fix(core): ignore RSL path guesses that return no RSL document
Random root .txt paths that 301 to the homepage were reported as an SPA catch-all, because the audit counted the homepage's final 200. A probe whose final path differs from the requested one now counts as absent. An HTML 200 at the requested path still fails.
fix(core): read a redirected root .txt probe as absent
A readable storefront that loads the Turnstile api.js for its forms still drew three findings. cart-handoff-reachability now fails only a cart that is a challenge page or only a widget. no-bot-detection passes a loader on a page the scan could read. no-blocking-captcha matches vendor script src and widget elements instead of the vendor name, so inline config, CSP lists and prose no longer match.
fix(core): stop reading a form Turnstile loader as a bot wall
The semantic extractor took the first <article>. On a page with many, that was a promo strip, and extraction determinism reported 0.00 agreement. It now takes the article with the most text, and falls back to the body when none holds a fifth of the page.
…article fix(core): read the largest article in the semantic extractor
The aggressive extractor dropped any element whose class contained a blocklisted word, which deleted a page wrapper named page-content--banner-enabled. It now tests whole class tokens and ids, and keeps an element holding most of the page. Script contents no longer count as prose, so JSON-LD URLs stop being key spans.
…tokens fix(core): match stripper class tokens, not substrings
The span after an h2/h3 was whatever text followed it. A tile grid became one answer crossing every block, and a one-word button repeated under each tile became an ambiguous answer. Heading spans now have to be sentences of three or more words; a sentence split across sibling blocks is still reported.
fix(core): take only sentences as text-fragment answer spans
Image arrays and ImageObjects were read as no image. A ProductGroup's AggregateOffer range and variant offer prices were read as no price, and productGroupID did not count as an item group. Resolve seller @id references on the page; a dangling reference still fails.
fix(core): read every JSON-LD shape in feed parity
A store locator's 'City or postcode' box matched the postal-code keyword and failed the audit. Query forms take a keyword or place, not profile data. Skip them, and skip location fields whose name offers a choice between concepts.
fix(core): skip search and locator forms in autofill coverage
Page builders emit whitespace-only <section> row slots. They have nothing to label and dragged a homepage to fail. Skip them, and return not applicable when no section has content, as the dossier requires.
fix(core): skip empty layout sections in section-headings
A feed is optional and the dossier says the audit informs rather than penalises. Return not applicable when no feed answers and no page advertises one. An advertised feed that does not answer still fails.
fix(core): rss-feed is not applicable without a feed
…ectioned fix(core): skip values, chrome and hidden text in fake-headings
…ards fix(core): match text fragments over nearest-block runs
fix(core): skip @id references in JSON-LD duplicate count
fix(core): read namespaced atom:link in feed WebSub discovery
…control fix(core): skip wrappers and inner labels of native controls in ghost scan
…e-only fix(core): compare aria-label with visible text on controls only
fix(core): treat FAQPage entries as editorial in UGC boundary scan
fix(core): decline contact-form when a contact page was not sampled
fix(core): skip hidden regions and keycaps in semantic-lists
…tional fix(core): report THIN sections without warning on them
…hidden fix(core): count title-kept h1 and skip hidden spans in survival recall
fix(core): let Last-Modified corroborate but not contradict lastmod
k2kirov
changed the title
chore(release): mark the false-positive sweep as 6.0.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
False-positive fixes found by scanning a large retail site, then a small marketing site. Each fix landed in this branch through its own PR, with failing tests first, a dossier update and a patch changeset. No audit id, grade, tier, weight or schema changes.
Merged fix PRs
*allow; missing robots.txt isnotApplicablenotApplicableproductGroupIDand seller@idnotApplicablenotApplicabledisplaydeclaration per selectorRe-scan of the same site, before and after
Overall score: 60 to 76.
Count-only changes, verdict unchanged (other real defects remain):
@idand no target-country signalNo other verdict moved.
Checks
Every fix PR ran the full gate: build, test, typecheck, lint, check:dossiers, check:requires, check:audit-map, format:check. Last run: 5585 tests passed.
Round 2: a small marketing site
A second report listed 12 suspected false positives. Each was checked against the audit code and its dossier; 10 held fully or in part, and each fix below landed through its own PR.
hiddenFromReaders()helper.notRendered(). One corpus page moves pass→fail: its FAQ answers sit only indisplay:nonepanels, which the matcher does not search.{"@id"}references are not duplicate declarations<atom:link>in RSS; ignore entry-level self linksnotApplicablewhen a sampled page links to an unscanned contact pagekbdkeycapsLast-Modifiedcorroborates but cannot contradictRejected after review (no change): CSS
display:blockon spans (no layout engine, documented), token-ratio's Readability numerator (documented),sitemap-lastmodcoverage (Bing documents lastmod as a recrawl signal, so omission is a real cost). Recorded under Deferred: hidden text in the shared semantic/density extractors; section-split's Evidence list cites unrelated sources.Re-scan of the marketing site, before and after round 2
Overall score: 93 to 97.
Count-only changes: survival recall 12/29 to 8/26 spans lost (the rest are visible homepage sections Readability drops); semantic-lists 3 to 1 pseudo-list; websub now reports only the missing hub. No other verdict moved.
Last full gate: 5638 tests passed, plus typecheck, lint, check:dossiers, check:requires, check:audit-map and format:check.