feat: expose per-check alert counts to metric formulas - #1166
Open
theredspoon wants to merge 1 commit into
Open
Conversation
AddAlert increments a per-check counter in a dedicated File.checkCounts field every time an alert is actually appended. That counter is exposed to metric formulas as an indexable Tengo object: check["Style.Rule"]. A check that's loaded and applicable but hasn't fired yet reads as 0; a name that doesn't correspond to a real, applicable check produces a real error naming it. Which checks are known and applicable is computed per file, respecting the same extension/section/style-toggling rules that already decide whether a check runs at all. Per-check counts live in their own field, decoupled from f.Metrics (which document content can influence via HTML tag-name bookkeeping), so a crafted document can't forge a count for a real check. An earlier version of this branch exposed counters as sanitized Tengo identifiers (check_Style_Rule), which needed to detect when two differently-named checks sanitized to the same identifier via a hand-rolled AST scope resolver. That approach kept finding new scope- tracking edge cases with no way to know when it was complete, and made a misspelled check name silently evaluate to zero instead of erroring. The indexable-object design removes the identifier-flattening step this was rooted in, so there's nothing left to collide. Adds an internal/e2e case covering the check[...] syntax end to end, verified against the pre-feature commit to confirm it fails there with a real Tengo compile error before passing cleanly here.
theredspoon
force-pushed
the
feat/metric-check-counts
branch
from
September 2, 2026 04:09
49624d6 to
c2ab873
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1163
Problem
metricrules already support arbitrary custom Tengo formulas, not just built-in readability formulas. What's missing is an input: there's no way for ametricformula to know how many times other checks have already fired, so a rule combining signals from several other rules isn't expressible.Motivating case
A style package like
tbhb/vale-ai-tellsincludes about 110 independent pattern rules, each an isolated per-instance match. One or two firing in a document reads as normal writing. Many different ones firing across the same document is a stronger signal. This makes that signal expressible as an ordinarymetricrule, socheck["AITells.FigurativeOwns"] + check["AITells.HedgingPhrases"] + check["AITells.EmptyPadding"] > 3becomes possible.Fix
Contained to
internal/core/file.go,internal/check/metric.go, a newinternal/check/check_counts.go,internal/lint/lint.go, and a small simplification incmd/vale/command.go'sls-metricscommand:AddAlertincrements a per-check counter in a dedicatedFile.checkCountsfield every time an alert is actually appended, so it only counts alerts that were actually reported, not deduped or hidden ones. This field is decoupled fromf.Metrics, which document content can influence via HTML tag-name bookkeeping, so nothing in the document itself can forge a count for a real check.metricformulas as an indexable object:check["Style.Rule"] + check["Style.OtherRule"] > 3.0until it actually fires.limit:/f.limitsbehavior is unaffected. Both mechanisms coexist, gated through the samef.Alertsappend.Counters are exposed as an indexable object (
check["Style.Rule"]) rather than as individually sanitized identifiers (check_Style_Rule), since check names come from user-chosen style and rule filenames and can contain arbitrary characters.Why indexing instead of sanitized identifiers
Flattening check names into identifiers has two problems:
An indexable object avoids both:
Performance
AddAlert's counter increment is oneO(1)map write.check[...]lookups each resolve to one map read, with no parsing.LoadedChecksis built once per file: a loop over every loaded check name, with a couple of map lookups and one short scan overf.BaseStylesper check. Benchmarked against a synthetic 110-rule style shaped like the motivating case, linting a realistic ~5KB document:LoadedChecksconstruction takes about 4µs, against about 550µs for the full lint pass, roughly 0.7% of total time.ComputeMetricsmakes one pass overf.Metrics.Testing
Coverage includes:
metricformula combining counts from two checksMinAlertLeveland via the--minAlertLevelCLI flaginternal/e2ecase (checks/metric/check-counts) exercisingcheck[...]end to end through a real.vale.ini/style/document, verified against the pre-feature commit to confirm it fails there with a real Tengo compile errorFull repo suite and
-raceare clean, includinginternal/e2e.