Fix three masking leaks (read rules, value cap, regex case)

CHANGELOG.md

@@ -33,15 +33,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 2. **`Mask(string)` never throws.** It runs the same UTF-8 path as the bytes API and produces the same output for the same text: comments are skipped (never written), trailing commas are accepted, non-ASCII and HTML characters are written unescaped (`RelaxedEscaping`), invalid or cut-off text yields its masked prefix. `Mask(string, out MaskResult, options)` reports the status. The `JsonReaderOptions`, `JsonWriterOptions`, `ignoreNulls` and `ignoreComments` parameters are gone: use `JsonObserverOptions` (`IgnoreNulls`, `Indented`, `MaxDepth`).
 3. **`Read(...)` never throws and returns a `MaskResult`** instead of `void`; `Read(byte[])` became `Read(ReadOnlySpan<byte>)`.
 4. **Every `Mask*` rule masks the whole value whatever its JSON type.** `MaskStr`, `MaskRawValue`, `MaskInt`, `MaskLong`, `MaskDecimal` and `MaskBool` no longer pass a value of another type to the default policy (where `BlockList` exposed it), and no longer descend into an object or array: a container is skipped unread and the function receives `null`. `MaskStr` hands a number or boolean to its function as its literal (`"12.50"`, `"true"`). As these rules now match containers too, a rule written before an `Obj(...)` or `Array(...)` rule for the same name takes precedence over it.
-5. **A string cut by `MaxValueBytes` reports `Truncated`** (with `FailedAtByte` `-1`), not `Masked`. A masking function receives a value longer than `MaxValueBytes` cut to that length.
-6. **Number read rules no longer fail the body:** `ReadInt`, `ReadLong` and `ReadDecimal` receive `null` for a number that does not fit the type, and the token is written unchanged.
+5. **A string cut by `MaxValueBytes` reports `Truncated`** (with `FailedAtByte` `-1`), not `Masked`. The cap applies to values written unmasked only (see 12).
+6. **Number read rules no longer fail the body:** `ReadInt`, `ReadLong` and `ReadDecimal` receive `null` for a number that does not fit the type; the default policy writes the token (see 11).
 7. **`PropertyPath` is a `ref struct` valid only during the call** it is passed to: `GetPropertyName`, `GetPropertyNameReverse`, `Length` and `ToString` remain; its constructor, `MaxLength` and `Dispose` are internal. Custom rules compiled against 1.x must be rebuilt. `ToString` writes array items as `[index]` (`a.b[0].c`, formerly `a.b..c`).
 8. **`JsonWriter` can no longer be derived from outside the library**, `JsonWriter.FromUtf8JsonWriter` and `JsonWriter.Empty` are removed, and `WriteCommentValue` is gone (comments are never written).
 9. **Internal now:** `JsonObserverException`, `PropertyPathMatch`, `JsonPropertyMatchDelegate`, `JsonPropertyPathMatchDelegate`, and the constructors of `JsonObjBuilder`, `JsonArrayBuilder`, `JsonValuePolicyBuilder` and their rule builders (start rules with `Match`).
 10. **A UTF-8 byte order mark at the start of the input is skipped.**
+11. **Read rules no longer decide what is written.** `ReadStr`, `ReadInt`, `ReadLong`, `ReadDecimal`, `ReadBool` and `ReadRaw` hand the value to the context; the next rule on the same match or the default policy writes it, so under `AllowList` a read value is `"***"`, not clear text. Chain `.Unmasked()` to keep it clear, or a mask method to read and mask one match: `Match("ssn").ReadStr(f).MaskAny(MaskTag.Last4)`. A read rule runs wherever it stands among the rules, so `Match("ssn").MaskAny(…).Match("ssn").ReadStr(f)` reads too. `Explain` reports `Read` only for a value read and written unchanged.
+12. **A masking function receives the whole value, and mask output is never cut by `MaxValueBytes`.** A `Last4`-style function sees the real last characters (`***4444`, not the ones at the cap), and `MaskTag.Hash`, a function's result and the `AllowList` stars are written whole, with status `Masked`. A function therefore decodes a long value in full.
 
 ### Fixed
 
+- **`PropMatches.Regex` follows `PropertyNameCaseInsensitive`:** under the default case-insensitive option it now also matches names that differ in case only (`DRiverLicensE` for `^driverLicense$`), as every other matcher does; a `Regex` built with `RegexOptions.IgnoreCase` ignores case under either option. A custom name test can follow the option through the new `PropMatchingStrategy(Func<string?, StringComparison, bool>)` constructor.
+
 - **Rules of an `Obj(...)` inside a property's `Array(...)` now apply** at any depth (`root.Match("lines").Array(l => l.Obj(…))`, and `Array(a => a.Array(b => b.Obj(…)))`). They looked for their names one or more levels too deep, so under `BlockList` those values were written in clear and under `AllowList` the whole item was masked. Only an `Obj(...)` directly under a root `Array(...)` worked.
 - **Relative rules receive `null` like absolute ones:** `MaskStr`, `MaskRawValue`, `MaskInt`, `MaskLong`, `MaskDecimal`, `MaskBool` and the read rules inside `Relative(...)` are called for a JSON `null`, as the rule-kinds table documents; `MaskAny` and `MaskAny(MaskTag)` keep `null` without calling the function.
 

DragoAnt.System.Text.Json.Observer.Tests.Shared/ApiSurfaceTests.cs

@@ -40,12 +40,12 @@ public void RelativeBuilder_EveryReadRule_KeepsValues()
     {
         var context = new Values();
         var observer = JsonObserver.Obj(JsonObserverValuePolicies<Values>.Relative(b => b
-                .Match("s").ReadStr((v, c) => c.Str = v)
-                .Match("i").ReadInt((v, c) => c.Int = v)
-                .Match("l").ReadLong((v, c) => c.Long = v)
-                .Match("d").ReadDecimal((v, c) => c.Decimal = v)
-                .Match("b").ReadBool((v, c) => c.Bool = v)
-                .Match("r").ReadRaw((v, c) => c.Raw = v)
+                .Match("s").ReadStr((v, c) => c.Str = v).Unmasked()
+                .Match("i").ReadInt((v, c) => c.Int = v).Unmasked()
+                .Match("l").ReadLong((v, c) => c.Long = v).Unmasked()
+                .Match("d").ReadDecimal((v, c) => c.Decimal = v).Unmasked()
+                .Match("b").ReadBool((v, c) => c.Bool = v).Unmasked()
+                .Match("r").ReadRaw((v, c) => c.Raw = v).Unmasked()
                 .Match("m").MaskInt((_, _) => "i")
                 .Match("n").MaskLong((_, _) => "l")
                 .Match("o").MaskDecimal((_, _) => "d")

DragoAnt.System.Text.Json.Observer.Tests.Shared/LeakTests.cs

@@ -0,0 +1,270 @@
+using System.Buffers;
+using System.Text;
+using System.Text.RegularExpressions;
+using DragoAnt.System.Text.Json.Observer.Strategies;
+
+namespace DragoAnt.System.Text.Json.Observer.Tests.Shared;
+
+public abstract class LeakTests
+{
+    private const string Ssn = """{"ssn":"123-45-6789","x":"y"}""";
+
+    private static JsonObserverValueDelegate<Holder> AllowList => JsonObserverValuePolicies<Holder>.AllowList;
+    private static JsonObserverValueDelegate<Holder> BlockList => JsonObserverValuePolicies<Holder>.BlockList;
+    private static JsonObserverValueDelegate<Holder> NullList => JsonObserverValuePolicies<Holder>.NullList;
+
+    [Fact]
+    public void Read_UnderAllowList_WritesTheValueMasked()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").ReadStr(Keep), AllowList).Mask(Ssn, holder)
+            .Should().Be("""{"ssn":"***","x":"***"}""");
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void Read_UnderBlockList_WritesTheValueUnchanged()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").ReadStr(Keep), BlockList).Mask(Ssn, holder)
+            .Should().Be(Ssn);
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void Read_UnderNullList_WritesNull()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").ReadStr(Keep), NullList).Mask(Ssn, holder)
+            .Should().Be("""{"ssn":null,"x":null}""");
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void Read_InRelativePolicy_UnderAllowList_WritesTheValueMasked()
+    {
+        var holder = new Holder();
+        var observer = JsonObserver.Obj<Holder>(JsonObserverValuePolicies<Holder>.Relative(b => b.Match("ssn").ReadStr(Keep)));
+
+        observer.Mask("""{"person":{"ssn":"123-45-6789"}}""", holder).Should().Be("""{"person":{"ssn":"***"}}""");
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void ReadNumber_UnderAllowList_WritesTheValueMasked()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Obj<Holder>(r => r.Match("pin").ReadInt((v, h) => h.Number = v)).Mask("""{"pin":1234}""", holder)
+            .Should().Be("""{"pin":"***"}""");
+        holder.Number.Should().Be(1234);
+    }
+
+    [Fact]
+    public void Read_ThenUnmasked_WritesClearText()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").ReadStr(Keep).Unmasked(), AllowList).Mask(Ssn, holder)
+            .Should().Be("""{"ssn":"123-45-6789","x":"***"}""");
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void Read_ThenMask_OnOneMatch_ReadsAndMasks()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").ReadStr(Keep).MaskAny(MaskTag.Last4), BlockList).Mask(Ssn, holder)
+            .Should().Be("""{"ssn":"***6789","x":"y"}""");
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void Mask_ThenRead_OnOneMatch_ReadsAndMasks()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").MaskAny(MaskTag.Full).Match("ssn").ReadStr(Keep), BlockList).Mask(Ssn, holder)
+            .Should().Be("""{"ssn":"***","x":"y"}""");
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void Read_ThenMask_InRelativePolicy_ReadsAndMasks()
+    {
+        var holder = new Holder();
+        var observer = JsonObserver.Obj<Holder>(JsonObserverValuePolicies<Holder>.Relative(
+            b => b.Match("ssn").ReadStr(Keep).MaskAny(MaskTag.Full), BlockList));
+
+        observer.Mask("""{"person":{"ssn":"123-45-6789","x":"y"}}""", holder).Should().Be("""{"person":{"ssn":"***","x":"y"}}""");
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void ArrayRead_UnderAllowList_WritesTheItemMasked()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Array<Holder>(a => a.ReadStr(Keep)).Mask("""["123-45-6789"]""", holder).Should().Be("""["***"]""");
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void ArrayRead_ThenUnmaskedOrMask_DecidesTheItem()
+    {
+        var unmasked = new Holder();
+        var masked = new Holder();
+
+        JsonObserver.Array<Holder>(a => a.ReadStr(Keep).Unmasked()).Mask("""["123-45-6789"]""", unmasked).Should().Be("""["123-45-6789"]""");
+        JsonObserver.Array<Holder>(a => a.MaskAny(MaskTag.Last4).ReadStr(Keep), BlockList).Mask("""["123-45-6789"]""", masked)
+            .Should().Be("""["***6789"]""");
+        unmasked.Value.Should().Be("123-45-6789");
+        masked.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void Read_ExtractionOnly_StillReads()
+    {
+        var holder = new Holder();
+
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").MaskAny(MaskTag.Full).Match("ssn").ReadStr(Keep)).Read(Ssn, holder)
+            .Status.Should().Be(MaskStatus.Masked);
+        holder.Value.Should().Be("123-45-6789");
+    }
+
+    [Fact]
+    public void Explain_Read_ReportsWhatIsWritten()
+    {
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").ReadStr(Keep), AllowList).Explain("ssn").Outcome
+            .Should().Be(JsonPathOutcome.Masked);
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").ReadStr(Keep).Unmasked(), AllowList).Explain("ssn").Outcome
+            .Should().Be(JsonPathOutcome.Read);
+        JsonObserver.Obj<Holder>(r => r.Match("ssn").ReadStr(Keep).MaskAny(MaskTag.Full), BlockList).Explain("ssn").Outcome
+            .Should().Be(JsonPathOutcome.Masked);
+    }
+
+    public static TheoryData<string> MaskFunctionKinds => ["MaskAny", "MaskStr", "MaskRawValue"];
+
+    [Theory]
+    [MemberData(nameof(MaskFunctionKinds))]
+    public void MaskFunction_UnderValueCap_ReceivesTheWholeValue(string kind)
+    {
+        var observer = JsonObserver.Obj(r => _ = kind switch
+        {
+            "MaskAny" => r.Match("card").MaskAny((s, _) => "***" + s![^4..]),
+            "MaskStr" => r.Match("card").MaskStr((s, _) => "***" + s![^4..]),
+            _ => r.Match("card").MaskRawValue((s, _) => "***" + s![^4..]),
+        });
+
+        observer.Mask("""{"card":"1111222233334444"}""", out var result, new JsonObserverOptions(MaxValueBytes: 8))
+            .Should().Be("""{"card":"***4444"}""");
+        result.Status.Should().Be(MaskStatus.Masked);
+    }
+
+    [Fact]
+    public void MaskFunction_UnderValueCap_ReceivesTheWholeValue_FromSegments()
+    {
+        var observer = JsonObserver.Obj(r => r.Match("card").MaskAny((s, _) => "***" + s![^4..]));
+        var output = new ArrayBufferWriter<byte>();
+
+        var result = observer.Mask(SequenceInputTests.Split("""{"card":"1111222233334444"}"""u8.ToArray(), 3), output,
+            new JsonObserverOptions(MaxValueBytes: 8));
+
+        Encoding.UTF8.GetString(output.WrittenSpan).Should().Be("""{"card":"***4444"}""");
+        result.Status.Should().Be(MaskStatus.Masked);
+    }
+
+    [Fact]
+    public void MaskOutput_LongerThanValueCap_IsNotCut()
+    {
+        var observer = JsonObserver.Obj(r => r.Match("card").MaskAny((_, _) => "replaced-by-a-long-mask"));
+
+        observer.Mask("""{"card":"1"}""", out var result, new JsonObserverOptions(MaxValueBytes: 8))
+            .Should().Be("""{"card":"replaced-by-a-long-mask"}""");
+        result.Status.Should().Be(MaskStatus.Masked);
+    }
+
+    [Fact]
+    public void Hash_UnderValueCap_IsNotCut()
+    {
+        var observer = JsonObserver.Obj(r => r.Match("card").MaskAny(MaskTag.Hash));
+        var options = new JsonObserverOptions(MaxValueBytes: 8, HashKey: "0123456789abcdef0123456789abcdef"u8.ToArray());
+
+        var masked = observer.Mask("""{"card":"1111222233334444"}""", out var result, options)!;
+        var uncapped = observer.Mask("""{"card":"1111222233334444"}""", options with { MaxValueBytes = int.MaxValue });
+
+        masked.Should().Be(uncapped);
+        JsonDocument.Parse(masked).RootElement.GetProperty("card").GetString().Should().StartWith("hash:").And.NotContain("…");
+        result.Status.Should().Be(MaskStatus.Masked);
+    }
+
+    [Fact]
+    public void UnmaskedValue_UnderValueCap_IsStillCut()
+    {
+        JsonObserver.Obj(JsonObserverValuePolicies.BlockList)
+            .Mask("""{"note":"1111222233334444"}""", out var result, new JsonObserverOptions(MaxValueBytes: 8))
+            .Should().Be("""{"note":"11112222…"}""");
+        result.Status.Should().Be(MaskStatus.Truncated);
+    }
+
+    private const string CaseCorpus =
+        """{"driverLicense":"A1","DRiverLicensE":"A2","driverlicense":"A3","driverLicense":"A4","drіverLicense":"A5","drİverLicense":"A6"}""";
+
+    public static TheoryData<string, bool, string> CaseTruthTable => new()
+    {
+        { "Match", true, "A1 A2 A3 A4" },
+        { "Match", false, "A1 A4" },
+        { "Regex", true, "A1 A2 A3 A4" },
+        { "Regex", false, "A1 A4" },
+        { "Function", true, "A1 A2 A3 A4" },
+        { "Function", false, "A1 A4" },
+        { "RelativeRegex", true, "A1 A2 A3 A4" },
+        { "RelativeRegex", false, "A1 A4" },
+    };
+
+    [Theory]
+    [MemberData(nameof(CaseTruthTable))]
+    public void CaseOption_ReachesEveryMatcher(string matcher, bool caseInsensitive, string expectedMasked)
+    {
+        PropMatchingStrategy match = matcher switch
+        {
+            "Match" => "driverLicense",
+            "Function" => new PropMatchingStrategy((name, comparison) => string.Equals(name, "driverLicense", comparison)),
+            _ => PropMatches.Regex(new Regex("^driverLicense$")),
+        };
+        var observer = matcher == "RelativeRegex"
+            ? JsonObserver.Obj(JsonObserverValuePolicies.Relative(b => b.Match(match).MaskAny(MaskTag.Full), JsonObserverValuePolicies.BlockList))
+            : JsonObserver.Obj(r => r.Match(match).MaskAny(MaskTag.Full), JsonObserverValuePolicies.BlockList);
+
+        var output = observer.Mask(CaseCorpus, new JsonObserverOptions(PropertyNameCaseInsensitive: caseInsensitive))!;
+
+        var masked = JsonDocument.Parse(output).RootElement.EnumerateObject()
+            .Select((p, i) => (Key: $"A{i + 1}", Value: p.Value.GetString()))
+            .Where(p => p.Value == "***")
+            .Select(p => p.Key);
+        string.Join(' ', masked).Should().Be(expectedMasked);
+    }
+
+    [Fact]
+    public void Regex_WithExplicitIgnoreCase_StaysCaseInsensitive()
+    {
+        var observer = JsonObserver.Obj(
+            r => r.Match(PropMatches.Regex(new Regex("^token$", RegexOptions.IgnoreCase))).MaskAny(MaskTag.Full),
+            JsonObserverValuePolicies.BlockList);
+
+        observer.Mask("""{"Token":"a"}""", new JsonObserverOptions(PropertyNameCaseInsensitive: false)).Should().Be("""{"Token":"***"}""");
+    }
+
+    private static void Keep(string? value, Holder holder) => holder.Value = value;
+
+    public sealed class Holder
+    {
+        public string? Value { get; set; }
+
+        public int? Number { get; set; }
+    }
+}

DragoAnt.System.Text.Json.Observer.Tests.Shared/LimitsTests.cs

@@ -26,44 +26,40 @@ public void MaxValueBytes_NoValueCut_ReportsMasked()
         => BytesApiTests.Mask(Observer, """{"note":"abcde"}""", new JsonObserverOptions(MaxValueBytes: 5)).Result.Status.Should().Be(MaskStatus.Masked);
 
     [Fact]
-    public void MaxValueBytes_StrategyOutputCut_ReportsTruncated()
+    public void MaxValueBytes_StrategyOutput_IsNotCut()
     {
         var observer = JsonObserver.Obj(b => b.Match("a").MaskStr((_, _) => new string('x', 100)), BlockList);
 
         var (result, output) = BytesApiTests.Mask(observer, """{"a":"v"}""", new JsonObserverOptions(MaxValueBytes: 4));
 
-        result.Status.Should().Be(MaskStatus.Truncated);
-        output.Should().Be("{\"a\":\"xxxx…\"}");
+        result.Status.Should().Be(MaskStatus.Masked);
+        output.Should().Be($"{{\"a\":\"{new string('x', 100)}\"}}");
     }
 
     [Fact]
-    public void MaxValueBytes_LongStrategyString_SurrogatePairNotSplit()
+    public void MaxValueBytes_LongCustomRuleString_SurrogatePairNotSplit()
     {
-        var observer = JsonObserver.Obj(b => b.Match("a").MaskStr((_, _) => "ab\U0001F600cd"), BlockList);
+        var observer = JsonObserver.Obj(
+            b => b.Match("a").MaskValue((ref Utf8JsonReader _, JsonWriter w, JsonObserveringEmptyContext _, ref PropertyPath _) => w.WriteStringValue("ab\U0001F600cd")),
+            BlockList);
 
         var (_, output) = BytesApiTests.Mask(observer, """{"a":"v"}""", new JsonObserverOptions(MaxValueBytes: 4));
 
         JsonDocument.Parse(output).RootElement.GetProperty("a").GetString().Should().Be("ab…");
     }
 
     [Fact]
-    public void MaskAny_Strategy_HugeValue_DoesNotDecodeWhole()
+    public void MaskAny_Strategy_HugeValue_ReceivesTheWholeValue()
     {
         var secret = new string('s', 5 * 1024 * 1024);
         var utf8 = Encoding.UTF8.GetBytes($$"""{"password":"{{secret}}","n":1}""");
         var observer = JsonObserver.Obj(Relative(b => b.Match("password").MaskAny((v, _) => v is null ? null : "len:" + v.Length), BlockList));
-        var options = new JsonObserverOptions(MaxValueBytes: 256);
         var output = new ArrayBufferWriter<byte>(1024);
-        observer.Mask(utf8, output, options);
-        output.Clear();
 
-        var before = GC.GetAllocatedBytesForCurrentThread();
-        var result = observer.Mask(utf8, output, options);
-        var allocated = GC.GetAllocatedBytesForCurrentThread() - before;
+        var result = observer.Mask(utf8, output, new JsonObserverOptions(MaxValueBytes: 256));
 
         result.Status.Should().Be(MaskStatus.Masked);
-        Encoding.UTF8.GetString(output.WrittenSpan).Should().Be("""{"password":"len:256","n":1}""");
-        allocated.Should().BeLessThan(64 * 1024);
+        Encoding.UTF8.GetString(output.WrittenSpan).Should().Be($$"""{"password":"len:{{secret.Length}}","n":1}""");
     }
 
     [Fact]

DragoAnt.System.Text.Json.Observer.Tests/RunLeakTests.cs

@@ -0,0 +1,3 @@
+namespace DragoAnt.System.Text.Json.Observer.Tests;
+
+public sealed class RunLeakTests : Shared.LeakTests;