v3.0.0 release

.github/codeql/codeql-config.yml

@@ -1,3 +1,22 @@
+# CodeQL configuration for flipbit/tokenizer
+#
+# Suppressed rules:
+#
+#   cs/linq/missed-where — Suggests replacing foreach+if with .Where().
+#     This project uses manual loops in hot paths to avoid LINQ allocation
+#     overhead. The pattern is a deliberate performance choice, not an
+#     oversight.
+#
+#   cs/linq/missed-select — Suggests replacing foreach with .Select().
+#     Same rationale as missed-where: manual loops avoid LINQ allocations
+#     on performance-sensitive code paths.
+
 paths-ignore:
   - '**/obj/**'
   - '**/generated/**'
+
+query-filters:
+  - exclude:
+      id: cs/linq/missed-where
+  - exclude:
+      id: cs/linq/missed-select

ARCHITECTURE.md

@@ -63,7 +63,7 @@ var options = new TokenizerOptions()
 
 ## Async Path
 
-The core compilation and tokenization logic is synchronous. `Tokenizer` and `TemplateMatcher` expose async overloads (`CompileAsync`, `TokenizeAsync`) for stream/reader-based I/O. The async path uses cooperative buffer refills via `TokenEnumerator.FillBufferAsync`, allowing tokenization of inputs larger than memory.
+The core compilation and tokenization logic is synchronous. `Tokenizer` and `TemplateMatcher` expose async overloads (`CompileAsync`, `TokenizeAsync`) for stream/reader-based I/O, using cooperative buffer refills via `TokenEnumerator.FillBufferAsync`. This allows tokenization of inputs larger than memory.
 
 ## Entry Points
 
@@ -76,16 +76,16 @@ Both are available via DI using `services.AddTokenizer()`.
 
 ## Diagnostics Subsystem
 
-The diagnostics subsystem provides structured tracing at two levels: **compilation** (template construction) and **tokenization** (runtime matching). It is opt-in via `TokenizerOptions.EnableDiagnostics` and uses null-object collectors to avoid any allocation overhead when disabled.
+The diagnostics subsystem provides structured tracing at two levels: **compilation** (template construction) and **tokenization** (runtime matching). It's opt-in via `TokenizerOptions.EnableDiagnostics` and uses null-object collectors to avoid any allocation overhead when disabled.
 
 ### Collectors
 
 Diagnostics are recorded through two collector interfaces, each with an active implementation and a no-op null singleton:
 
 | Interface | Active Implementation | Null Implementation | Scope |
 |-----------|----------------------|--------------------|----|
-| `ICompilationDiagnosticCollector` | `CompilationDiagnosticCollector` | `NullCompilationDiagnosticCollector` | Template construction — hints, tokens, decorators |
-| `ITokenizationDiagnosticCollector` | `TokenizationDiagnosticCollector` | `NullTokenizationDiagnosticCollector` | Runtime matching — preambles, values, validators |
+| `ICompilationDiagnosticCollector` | `CompilationDiagnosticCollector` | `NullCompilationDiagnosticCollector` | Template construction - hints, tokens, decorators |
+| `ITokenizationDiagnosticCollector` | `TokenizationDiagnosticCollector` | `NullTokenizationDiagnosticCollector` | Runtime matching - preambles, values, validators |
 
 Both expose `bool IsEnabled`, used as a guard at call sites to skip argument evaluation when diagnostics are off:
 
@@ -102,14 +102,14 @@ Events are stored as `DiagnosticEvent<TType>`, a generic container parameterised
 
 ```
 DiagnosticEvent<TType>
-├── TType Type           — event kind enum value
-├── string? TokenName    — token this event relates to
-├── int? TokenId         — unique token ID within template
-├── FileLocation? Location — position in input/source
-├── string? Value        — value being tested/assigned
-├── string? Detail       — human-readable explanation
-├── string? DecoratorName — validator/transformer name
-└── string[]? DecoratorArgs — decorator parameters
+├── TType Type           - event kind enum value
+├── string? TokenName    - token this event relates to
+├── int? TokenId         - unique token ID within template
+├── FileLocation? Location - position in input/source
+├── string? Value        - value being tested/assigned
+├── string? Detail       - human-readable explanation
+├── string? DecoratorName - validator/transformer name
+└── string[]? DecoratorArgs - decorator parameters
 ```
 
 Global type aliases simplify usage:
@@ -130,20 +130,20 @@ Raw events are transformed into a per-token diagnostic view by `TokenDiagnosticB
 ```
 TokenDiagnostic
 ├── TokenName, TokenId
-├── TokenOutcome         — Matched | Rejected | NeverFound | Blocked
-├── TokenAttempt[]       — every consideration during tokenization
+├── TokenOutcome         - Matched | Rejected | NeverFound | Blocked
+├── TokenAttempt[]       - every consideration during tokenization
 │   ├── Location, Value
-│   ├── AttemptOutcome   — Assigned | ValidatorRejected | TransformerFailed | Backtracked
+│   ├── AttemptOutcome   - Assigned | ValidatorRejected | TransformerFailed | Backtracked
 │   └── DecoratorName, Reason
-├── AssignedValues[]     — matched values (multiple for repeating tokens)
-├── AssignedLocations[]  — parallel to AssignedValues
-├── BlockedBy            — name of blocker token (Blocked outcome only)
-└── DiagnosticIssue[]    — problems with adaptive hints
-    ├── Code             — stable issue code (TK001–TK008)
+├── AssignedValues[]     - matched values (multiple for repeating tokens)
+├── AssignedLocations[]  - parallel to AssignedValues
+├── BlockedBy            - name of blocker token (Blocked outcome only)
+└── DiagnosticIssue[]    - problems with adaptive hints
+    ├── Code             - stable issue code (TK001–TK008)
     ├── DiagnosticIssueType
     ├── Description
     ├── Location
-    └── Hint             — contextual suggestion from hint generators
+    └── Hint             - contextual suggestion from hint generators
 ```
 
 The token-centric view is built lazily on first access and cached. Raw events remain available via `RawEvents` for low-level tracing.
@@ -152,13 +152,13 @@ The token-centric view is built lazily on first access and cached. Raw events re
 
 `TokenDiagnosticBuilder` transforms raw `TokenizationEvent` lists into `TokenDiagnostic` arrays through four ordered phases:
 
-1. **CollectEvents** — walks all raw events, builds per-token attempt lists, issue lists, assigned value entries, and context indexes for cross-referencing.
+1. **CollectEvents** - walks all raw events, builds per-token attempt lists, issue lists, assigned value entries, and context indexes for cross-referencing.
 
-2. **ClassifyOutcomes** — creates `TokenDiagnostic` objects. Determines each token's `TokenOutcome` based on whether it was assigned, rejected, or missed. Runs `ValueMismatch` detection (checks whether a matched token's value contains the preamble of a missed token, suggesting greedy capture).
+2. **ClassifyOutcomes** - creates `TokenDiagnostic` objects. Determines each token's `TokenOutcome` based on whether it was assigned, rejected, or missed. Runs `ValueMismatch` detection (checks whether a matched token's value contains the preamble of a missed token, suggesting greedy capture).
 
-3. **ApplyBlockedAnnotations** — causality analysis for ordered mode only. Finds the first non-optional unmatched token (the "blocker") and reclassifies subsequent `NeverFound` tokens as `Blocked`.
+3. **ApplyBlockedAnnotations** - causality analysis for ordered mode only. Finds the first non-optional unmatched token (the "blocker") and reclassifies subsequent `NeverFound` tokens as `Blocked`.
 
-4. **BuildVerdict** — generates a human-readable summary (e.g. "Matched 3 of 5 tokens (2 missed).").
+4. **BuildVerdict** - generates a human-readable summary (e.g. "Matched 3 of 5 tokens (2 missed).").
 
 ### Issue Codes
 
@@ -177,7 +177,7 @@ Each `DiagnosticIssue` carries a stable code from `IssueCodeMap` for programmati
 
 ### Hint Generators
 
-`IssueFactory` chains `IHintGenerator` implementations to produce contextual suggestions for each issue. All generators are stateless and shared via a static default factory. Each receives the source event and a `BuildContext` containing cross-token indexes (input lines, rejections per token, decorator successes, optional token names).
+`IssueFactory` chains `IHintGenerator` implementations to produce contextual suggestions for each issue. All generators are stateless and shared via a static default factory. Each one receives the source event and a `BuildContext` containing cross-token indexes (input lines, rejections per token, decorator successes, optional token names).
 
 | Generator | Detects | Example Hint |
 |-----------|---------|-------------|
@@ -186,18 +186,18 @@ Each `DiagnosticIssue` carries a stable code from `IssueCodeMap` for programmati
 | `DateFormatHintGenerator` | Failed date transformers, tries 18 common formats | "Value matches format 'dd/MM/yyyy'. Change transformer to use it." |
 | `ChainedDecoratorHintGenerator` | Prior decorator succeeded, next failed | "Decorator chain: 'Trim' succeeded → 'ToInt' rejected value 'abc'." |
 | `MultipleRejectionHintGenerator` | Token rejected 2+ times | "Token was rejected 3 times. Values tried: 'a', 'b', 'c'." |
-| `OptionalTokenHintGenerator` | Optional token not found | "Token 'MiddleName' is optional — no action needed." |
+| `OptionalTokenHintGenerator` | Optional token not found | "Token 'MiddleName' is optional - no action needed." |
 | `RepeatingTokenHintGenerator` | Repeating token disabled early | "Repeating token disabled. Value 'x' failed IsNumeric validation." |
 | `ValueMismatchHintGenerator` | Greedy capture swallowed another preamble | "Consider adding '$' to prevent greedy capture." |
-| `BlockedTokenHintGenerator` | Token blocked by prior failure | "Fix 'FirstName' first — this token may match once resolved." |
+| `BlockedTokenHintGenerator` | Token blocked by prior failure | "Fix 'FirstName' first - this token may match once resolved." |
 
 ### Renderers
 
 Two renderers produce human-readable diagnostic output:
 
-**AlignmentRenderer** (`AlignmentRenderer.Render`) — structured view of template-to-input mapping with sections for matched tokens, failures, unmatched tokens, and blocked tokens. Includes assigned values with line locations, decorator details, and hints.
+**AlignmentRenderer** (`AlignmentRenderer.Render`) - structured view of template-to-input mapping with sections for matched tokens, failures, unmatched tokens, and blocked tokens. Includes assigned values with line locations, decorator details, and hints.
 
-**ProcessingOrderRenderer** (`ProcessingOrderRenderer.Render`) — chronological walk-through of every engine decision, showing event type, token name, location, value, decorator, and detail for each step.
+**ProcessingOrderRenderer** (`ProcessingOrderRenderer.Render`) - chronological walk-through of every engine decision, showing event type, token name, location, value, decorator, and detail for each step.
 
 Both are called automatically during `FinalizeTokenization` when diagnostics are enabled, with alignment logged at Warning level and processing order at Debug level.
 

CHANGELOG.md

@@ -5,7 +5,51 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/),
 and this project adheres to [Semantic Versioning](https://semver.org/).
 
-## [3.0.0-beta.1] - 2026-08-08
+## [3.0.0] - 2026-07-23
+
+### Added
+
+- `Assign<T>(T target)` overload for populating existing object instances (classes, structs, records, and types without parameterless constructors)
+- Online documentation at pullpatchpush.com/tokenizer with interactive playground
+- Configuration reference and Extensibility guide on docs site
+
+### Changed
+
+- Library icon updated to Phosphor brackets-curly design
+- README trimmed to landing-page format with links to documentation site
+- README links converted to absolute URLs for NuGet rendering
+
+## [3.0.0-beta.2] - 2026-07-09
+
+### Added
+
+- Token-centric diagnostic model with per-token outcome tracking (Matched, Rejected, NeverFound, Blocked), match attempt history, and assigned value locations
+- Diagnostic hint generators: PreambleNearMiss, ValidatorValue, DateFormat, ChainedDecorator, MultipleRejection, OptionalToken, RepeatingToken, ValueMismatch, BlockedToken
+- Stable diagnostic issue codes (TK001–TK008) for programmatic filtering
+- AlignmentRenderer and ProcessingOrderRenderer for human-readable diagnostic output
+- Causality analysis for ordered-mode diagnostics (blocked token detection)
+- `MaxRegexTimeout` option on `TokenizerOptions` (default: 1 second) to bound regex evaluation in user-supplied patterns
+- `CancellationToken` overloads on synchronous `Tokenize` methods
+- SECURITY.md with guidance for processing untrusted input
+- 61 diagnostic characterisation tests
+
+### Changed
+
+- Diagnostic subsystem redesigned from flat event stream to token-centric model (`TokenDiagnostic`, `TokenAttempt`, `DiagnosticIssue`)
+- `DiagnosticResult` replaced with `TokenizationDiagnostics` (lazy-built token view, raw event access kept)
+- Singular `AssignedValue`/`AssignedLocation` on `TokenDiagnostic` replaced with list-based `AssignedValues`/`AssignedLocations` for repeating token support
+- Compilation and tokenization diagnostic collectors separated (`ICompilationDiagnosticCollector`, `ITokenizationDiagnosticCollector`)
+- `MatchesRegexValidator` hardened: catches `RegexMatchTimeoutException`, removed `RegexOptions.Compiled`, added bounded cache eviction
+- `RegexReplaceTransformer` hardened: catches timeout, uses `MaxRegexTimeout`
+- Removed unbounded static `PathSegmentCache` (replaced with instance-scoped caching)
+- PII logging: guarded exception messages at Debug level, downgraded diagnostic log output
+
+### Fixed
+
+- All CodeQL code scanning alerts resolved (catch-of-all-exceptions, useless-assignment, local-not-disposed, dispose-not-called-on-throw, useless-upcast, missed-readonly, nested-if, missed-ternary, null-argument-to-equals, path-combine, useless-gethashcode, misleading-indentation)
+- Infinite regex timeouts on netstandard2.0 fallback paths
+
+## [3.0.0-beta.1] - 2026-06-20
 
 ### Added
 

CONTRIBUTING.md

@@ -22,11 +22,11 @@ dotnet build ./Tokenizer.sln
 dotnet test ./tests/Tokenizer.Tests/Tokenizer.Tests.csproj
 ```
 
-All pull requests must pass the full test suite on both Ubuntu and Windows (enforced by CI).
+All pull requests need to pass the full test suite on both Ubuntu and Windows (CI enforces this).
 
 ## Code Style
 
-Code style is enforced by `.editorconfig` and Roslyn analyzers. The build will fail on violations. To check formatting:
+Code style is enforced by `.editorconfig` and Roslyn analyzers, so the build will fail on violations. To check formatting:
 
 ```bash
 dotnet format style ./Tokenizer.sln --verify-no-changes
@@ -43,7 +43,7 @@ dotnet format style ./Tokenizer.sln
 - Keep changes focused. One logical change per PR.
 - Add tests for new functionality and bug fixes.
 - Update `CHANGELOG.md` under the `[Unreleased]` section.
-- Ensure all tests pass and the build is clean before submitting.
+- Make sure all tests pass and the build is clean before submitting.
 
 ## Architecture