Skip to content

feat: expose per-check alert counts to metric formulas - #1166

Open
theredspoon wants to merge 1 commit into
vale-cli:v3from
theredspoon:feat/metric-check-counts
Open

feat: expose per-check alert counts to metric formulas#1166
theredspoon wants to merge 1 commit into
vale-cli:v3from
theredspoon:feat/metric-check-counts

Conversation

@theredspoon

@theredspoon theredspoon commented Sep 2, 2026

Copy link
Copy Markdown

Closes #1163

Problem

metric rules already support arbitrary custom Tengo formulas, not just built-in readability formulas. What's missing is an input: there's no way for a metric formula to know how many times other checks have already fired, so a rule combining signals from several other rules isn't expressible.

Motivating case

A style package like tbhb/vale-ai-tells includes about 110 independent pattern rules, each an isolated per-instance match. One or two firing in a document reads as normal writing. Many different ones firing across the same document is a stronger signal. This makes that signal expressible as an ordinary metric rule, so check["AITells.FigurativeOwns"] + check["AITells.HedgingPhrases"] + check["AITells.EmptyPadding"] > 3 becomes possible.

Fix

Contained to internal/core/file.go, internal/check/metric.go, a new internal/check/check_counts.go, internal/lint/lint.go, and a small simplification in cmd/vale/command.go's ls-metrics command:

  • AddAlert increments a per-check counter in a dedicated File.checkCounts field every time an alert is actually appended, so it only counts alerts that were actually reported, not deduped or hidden ones. This field is decoupled from f.Metrics, which document content can influence via HTML tag-name bookkeeping, so nothing in the document itself can forge a count for a real check.
  • That counter is exposed to metric formulas as an indexable object: check["Style.Rule"] + check["Style.OtherRule"] > 3.
    • A check that's loaded and applicable to the current file reads as 0 until it actually fires.
    • A name that doesn't correspond to a real, applicable check produces a clear error naming it, so a typo in a formula raises an error instead of silently evaluating as if the check simply hadn't fired.
  • Which checks are known and applicable is computed per file, respecting the same extension, section, and style-toggling rules that already decide whether a check runs at all, not just the full set of everything loaded anywhere in the config.
  • Existing opt-in limit:/f.limits behavior is unaffected. Both mechanisms coexist, gated through the same f.Alerts append.

Counters are exposed as an indexable object (check["Style.Rule"]) rather than as individually sanitized identifiers (check_Style_Rule), since check names come from user-chosen style and rule filenames and can contain arbitrary characters.

Why indexing instead of sanitized identifiers

Flattening check names into identifiers has two problems:

  • Two differently-named checks can collide on the same identifier.
  • Reliably knowing which identifiers a Tengo formula references requires resolving scope the same way Tengo itself does: function literals, local shadowing, nested blocks, and so on.

An indexable object avoids both:

  • There's no flattening step for two names to collide on.
  • A lookup checks against the real, currently-applicable set of check names directly, so an unknown name is a clear error rather than a silent zero.

Performance

AddAlert's counter increment is one O(1) map write. check[...] lookups each resolve to one map read, with no parsing.

LoadedChecks is built once per file: a loop over every loaded check name, with a couple of map lookups and one short scan over f.BaseStyles per check. Benchmarked against a synthetic 110-rule style shaped like the motivating case, linting a realistic ~5KB document: LoadedChecks construction takes about 4µs, against about 550µs for the full lint pass, roughly 0.7% of total time. ComputeMetrics makes one pass over f.Metrics.

Testing

Coverage includes:

  • the counter only incrementing for alerts that were actually reported
  • a real multi-rule metric formula combining counts from two checks
  • a missing-but-applicable check reading as zero, including under a lowered MinAlertLevel and via the --minAlertLevel CLI flag
  • an unknown check name producing a real error
  • two check names differing only in punctuation resolving correctly and independently
  • a regression test for the crafted-tag-injection case described above
  • an internal/e2e case (checks/metric/check-counts) exercising check[...] end to end through a real .vale.ini/style/document, verified against the pre-feature commit to confirm it fails there with a real Tengo compile error

Full repo suite and -race are clean, including internal/e2e.

@theredspoon theredspoon changed the title feat(metric): expose per-check alert counts to metric formulas via an indexable check[...] object feat(metric): expose per-check alert counts to formulas Sep 2, 2026
@theredspoon theredspoon changed the title feat(metric): expose per-check alert counts to formulas feat: expose per-check alert counts to metric formulas Sep 2, 2026
AddAlert increments a per-check counter in a dedicated File.checkCounts
field every time an alert is actually appended. That counter is exposed
to metric formulas as an indexable Tengo object: check["Style.Rule"].
A check that's loaded and applicable but hasn't fired yet reads as 0;
a name that doesn't correspond to a real, applicable check produces a
real error naming it. Which checks are known and applicable is computed
per file, respecting the same extension/section/style-toggling rules
that already decide whether a check runs at all.

Per-check counts live in their own field, decoupled from f.Metrics
(which document content can influence via HTML tag-name bookkeeping),
so a crafted document can't forge a count for a real check.

An earlier version of this branch exposed counters as sanitized Tengo
identifiers (check_Style_Rule), which needed to detect when two
differently-named checks sanitized to the same identifier via a
hand-rolled AST scope resolver. That approach kept finding new scope-
tracking edge cases with no way to know when it was complete, and made
a misspelled check name silently evaluate to zero instead of erroring.
The indexable-object design removes the identifier-flattening step
this was rooted in, so there's nothing left to collide.

Adds an internal/e2e case covering the check[...] syntax end to end,
verified against the pre-feature commit to confirm it fails there with
a real Tengo compile error before passing cleanly here.
@theredspoon
theredspoon force-pushed the feat/metric-check-counts branch from 49624d6 to c2ab873 Compare September 2, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose per-check alert counts to metric formulas, for composite/density scoring across multiple rules

1 participant