Observer 2.0 core API: nested-array fix, paths with indices, shape metadata, 0 B warm calls

CHANGELOG.md

@@ -16,6 +16,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **UTF-8 name matching:** property names are compared on their UTF-8 bytes, without creating strings.
 - **Verbatim pass-through:** unmasked numbers and strings are copied as written (`1.50`, `1e400`, `-0`, 20-digit integers).
 - `PropertyPath.Length`.
+- **`ReadOnlySequence<byte>` input:** `Mask(in ReadOnlySequence<byte>, …)` and `Read(in ReadOnlySequence<byte>, …)` mask a payload held in several buffers, for example from a `PipeReader`, without copying it into one; the output is byte for byte what the span overload writes, however the bytes are split.
+- **No per-call allocation on the bytes API:** the writers are reused per thread, so a warm `Mask`/`Read` of bytes with constant or tag rules allocates nothing (pinned by a test for the span, sequence, ignore-nulls and read paths).
+- **`Explain(path)`:** `JsonObserver.Explain("lines[0].qty", JsonTokenType.Number)` returns a `JsonPathExplanation` naming the rule or policy that handles the value, its action, the outcome (`Unchanged`, `Masked`, `Read`, `Custom`, `Invalid`) and one step per level, for rule-based and shape observers.
+- **Classified tags:** `MaskTag` carries an optional `Key` (a data classification, a redactor name) and `MaskKind.Custom`, so a strategy maps its own taxonomy without casting enum values; `MaskTag.Custom(key)`, `TryGetKey<T>`.
+- **Strategies see where a value is:** `Utf8MaskStrategy.Mask(in Utf8MaskContext, JsonWriter)` receives the value, its JSON type, the tag, the options, the property name and the whole path without allocating. Both `Mask` overloads are virtual; a strategy overrides the one it needs.
+- **`JsonWriter` span overloads:** `WriteStringValue(ReadOnlySpan<char>)`, `WritePropertyName(ReadOnlySpan<char>)`, `WriteBase64StringValue(ReadOnlySpan<byte>)` and `WriteNumberValue(double)`.
+- **Array indices in paths:** `PropertyPath.ToString()` renders `items[2].sku` (names that need it as `['a.b']`); `TryGetArrayIndex`, `IsArrayItem` and `TryGetPropertyNameUtf8` give zero-allocation access.
+- **Case sensitivity:** `JsonObserverOptions.PropertyNameCaseInsensitive` (default `true`) makes rules, `PropMatches` tests and shapes match names exactly when set to `false`, the way the serializer does; `JsonShapeOptions.PropertyNameCaseInsensitive` and `JsonShapeOptions.FromSerializerOptions(...)` set it for one shape observer; `PropertyPath.PropertyNameCaseInsensitive` tells a custom rule.
+- **Metadata on shapes:** `JsonShape.Members` lists `JsonShapeProperty` items with the `JsonPropertyInfo`, CLR member, property and declaring type, `IsRequired`, `IsNullable` and custom attributes; nodes carry their `JsonTypeInfo`/`ClrType`; nodes and properties have `Annotations` for integrations; `FromTypeInfo` takes an `annotate` callback, and `FindMember` looks a property up by its UTF-8 name. On .NET 8, source-generated metadata has no attributes or reference-type nullability.
 - **New package `DragoAnt.System.Text.Json.Observer.Http`:** `JsonBodyLoggingHandler` logs masked `HttpClient` request and response bodies; register it with `AddJsonBodyLogging`, pick maskers per body model type with `IJsonBodyMaskerProvider`, and attach model types per request with `WithBodyLogging<TRequest, TResponse>()`.
 
 ### Changed — breaking
@@ -26,11 +35,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 4. **Every `Mask*` rule masks the whole value whatever its JSON type.** `MaskStr`, `MaskRawValue`, `MaskInt`, `MaskLong`, `MaskDecimal` and `MaskBool` no longer pass a value of another type to the default policy (where `BlockList` exposed it), and no longer descend into an object or array: a container is skipped unread and the function receives `null`. `MaskStr` hands a number or boolean to its function as its literal (`"12.50"`, `"true"`). As these rules now match containers too, a rule written before an `Obj(...)` or `Array(...)` rule for the same name takes precedence over it.
 5. **A string cut by `MaxValueBytes` reports `Truncated`** (with `FailedAtByte` `-1`), not `Masked`. A masking function receives a value longer than `MaxValueBytes` cut to that length.
 6. **Number read rules no longer fail the body:** `ReadInt`, `ReadLong` and `ReadDecimal` receive `null` for a number that does not fit the type, and the token is written unchanged.
-7. **`PropertyPath` is a `ref struct` valid only during the call** it is passed to: `GetPropertyName`, `GetPropertyNameReverse`, `Length` and `ToString` remain; its constructor, `MaxLength` and `Dispose` are internal. Custom rules compiled against 1.x must be rebuilt.
+7. **`PropertyPath` is a `ref struct` valid only during the call** it is passed to: `GetPropertyName`, `GetPropertyNameReverse`, `Length` and `ToString` remain; its constructor, `MaxLength` and `Dispose` are internal. Custom rules compiled against 1.x must be rebuilt. `ToString` writes array items as `[index]` (`a.b[0].c`, formerly `a.b..c`).
 8. **`JsonWriter` can no longer be derived from outside the library**, `JsonWriter.FromUtf8JsonWriter` and `JsonWriter.Empty` are removed, and `WriteCommentValue` is gone (comments are never written).
 9. **Internal now:** `JsonObserverException`, `PropertyPathMatch`, `JsonPropertyMatchDelegate`, `JsonPropertyPathMatchDelegate`, and the constructors of `JsonObjBuilder`, `JsonArrayBuilder`, `JsonValuePolicyBuilder` and their rule builders (start rules with `Match`).
 10. **A UTF-8 byte order mark at the start of the input is skipped.**
 
+### Fixed
+
+- **Rules of an `Obj(...)` inside a property's `Array(...)` now apply** at any depth (`root.Match("lines").Array(l => l.Obj(…))`, and `Array(a => a.Array(b => b.Obj(…)))`). They looked for their names one or more levels too deep, so under `BlockList` those values were written in clear and under `AllowList` the whole item was masked. Only an `Obj(...)` directly under a root `Array(...)` worked.
+- **Relative rules receive `null` like absolute ones:** `MaskStr`, `MaskRawValue`, `MaskInt`, `MaskLong`, `MaskDecimal`, `MaskBool` and the read rules inside `Relative(...)` are called for a JSON `null`, as the rule-kinds table documents; `MaskAny` and `MaskAny(MaskTag)` keep `null` without calling the function.
+
 ### Observer.Http 2.0.0 — breaking (since the preview builds)
 
 - `JsonBodyOutcome.Timeout` is replaced by `Canceled`: every cancellation, an `HttpClient.Timeout` included, is logged as `Canceled`.

DragoAnt.System.Text.Json.Observer.Tests.Shared/AllocationTests.cs

@@ -11,40 +11,96 @@ public abstract class AllocationTests
         _ => { },
         Relative(b => b.Match("password").MaskAny("***").Match("card", "number").MaskStr("***"), BlockList));
 
-    public static TheoryData<string, int, long> Budgets => new()
+    private static readonly JsonObserver<JsonObserveringEmptyContext> Reader = JsonObserver.Any<JsonObserveringEmptyContext>(
+        _ => { },
+        _ => { },
+        JsonObserverValuePolicies<JsonObserveringEmptyContext>.Relative(
+            b => b.Match("password").MaskAny("***"),
+            JsonObserverValuePolicies<JsonObserveringEmptyContext>.BlockList));
+
+    private static readonly JsonObserverOptions IgnoreNulls = new(IgnoreNulls: true);
+
+    public static TheoryData<string, int, string> Budgets()
     {
-        { "flat", 1024, 512 },
-        { "flat", 64 * 1024, 512 },
-        { "nested", 8 * 1024, 512 },
-        { "array", 8 * 1024, 512 },
-    };
+        var data = new TheoryData<string, int, string>();
+        foreach (var api in new[] { "span", "sequence", "ignore-nulls", "read" })
+        {
+            data.Add("flat", 1024, api);
+            data.Add("flat", 64 * 1024, api);
+            data.Add("nested", 8 * 1024, api);
+            data.Add("array", 8 * 1024, api);
+        }
 
+        return data;
+    }
+
+    /// <summary>
+    /// The bytes API allocates nothing per call once warm: writers are reused per thread and buffers come from the pool.
+    /// </summary>
     [Theory]
     [MemberData(nameof(Budgets))]
-    public void BytesApi_ReusedOutput_StaysWithinBudget(string shape, int size, long budget)
+    public void BytesApi_ReusedOutput_AllocatesNothing(string shape, int size, string api)
     {
         var utf8 = Encoding.UTF8.GetBytes(Payload(shape, size));
+        var sequence = SequenceInputTests.Split(utf8, 4096);
         var output = new ArrayBufferWriter<byte>(utf8.Length * 2);
-        for (var i = 0; i < 20; i++)
+
+        void Call()
         {
             output.ResetWrittenCount();
-            Observer.Mask(utf8, output);
+            _ = api switch
+            {
+                "span" => Observer.Mask(utf8, output),
+                "sequence" => Observer.Mask(sequence, output),
+                "ignore-nulls" => Observer.Mask(utf8, output, IgnoreNulls),
+                _ => Reader.Read(utf8, JsonObserveringEmptyContext.Instance),
+            };
         }
 
+        for (var i = 0; i < 20; i++)
+        {
+            Call();
+        }
+
+        // A one-off runtime allocation (tier-up under a loaded test host) lands in one round; a real per-call cost lands in all.
         const int calls = 50;
-        var before = GC.GetAllocatedBytesForCurrentThread();
-        for (var i = 0; i < calls; i++)
+        var perCall = long.MaxValue;
+        for (var round = 0; round < 5 && perCall > 0; round++)
         {
-            output.ResetWrittenCount();
-            Observer.Mask(utf8, output);
+            var before = GC.GetAllocatedBytesForCurrentThread();
+            for (var i = 0; i < calls; i++)
+            {
+                Call();
+            }
+
+            perCall = Math.Min(perCall, (GC.GetAllocatedBytesForCurrentThread() - before) / calls);
         }
 
-        var perCall = (GC.GetAllocatedBytesForCurrentThread() - before) / calls;
+        perCall.Should().Be(0, $"{api} {shape} {size} B allocates {perCall} B per call");
+    }
+
+    [Fact]
+    public void NestedCallOnSameThread_GetsItsOwnWriter()
+    {
+        var inner = JsonObserver.Obj(Relative(b => b.Match("pin").MaskAny("#"), BlockList));
+        var outer = JsonObserver.Obj(b => b.Match("payload").MaskStr((v, _) => inner.Mask(v)), BlockList);
+
+        outer.Mask("""{"payload":"{\"pin\":1,\"x\":2}","y":3}""")
+            .Should().Be("""{"payload":"{\"pin\":\"#\",\"x\":2}","y":3}""");
+    }
+
+    [Fact]
+    public void WriterSettingsChange_BetweenCalls_Respected()
+    {
+        const string json = """{"a":"é<","password":"x"}""";
 
-        perCall.Should().BeLessThanOrEqualTo(budget, $"{shape} {size} B allocates {perCall} B per call");
+        Observer.Mask(json).Should().Be("""{"a":"é<","password":"***"}""");
+        Observer.Mask(json, new JsonObserverOptions(RelaxedEscaping: false)).Should().NotContain("é").And.Contain((char)92 + "u003C").And.EndWith(",\"password\":\"***\"}");
+        Observer.Mask(json, new JsonObserverOptions(Indented: true)).Should().Contain(Environment.NewLine);
+        Observer.Mask(json).Should().Be("""{"a":"é<","password":"***"}""");
     }
 
-    private static string Payload(string shape, int size)
+    internal static string Payload(string shape, int size)
     {
         var json = new StringBuilder();
         var i = 0;

DragoAnt.System.Text.Json.Observer.Tests.Shared/BytesApiTests.cs

@@ -8,12 +8,12 @@ public abstract class BytesApiTests
 {
     private const string Secret = "S3cr3tV4l";
 
-    private static readonly JsonObserver Observer = JsonObserver.Any(
+    internal static readonly JsonObserver Observer = JsonObserver.Any(
         _ => { },
         _ => { },
         Relative(b => b.Match("password").MaskAny("***").Match("pin").MaskAny("***"), BlockList));
 
-    private static readonly string[] Payloads =
+    internal static readonly string[] Payloads =
     [
         $$$"""{"user":"bob","password":"{{{Secret}}}","card":{"pin":"{{{Secret}}}","exp":"12/30"},"items":[{"id":1,"password":["{{{Secret}}}",{"x":"{{{Secret}}}"}]},{"id":2,"note":"ok"}],"active":true,"amount":1.5e3}""",
         $$$$$"""[{"password":{"value":"{{{{{Secret}}}}}","deep":[1,2,{"s":"{{{{{Secret}}}}}"}]}},null,"text",[{"pin":12345678}],{"a":{"b":{"c":{"password":"{{{{{Secret}}}}}"}}}}]""",

DragoAnt.System.Text.Json.Observer.Tests.Shared/CaseSensitivityTests.cs

@@ -0,0 +1,92 @@
+using System.Text;
+using System.Text.Json.Serialization.Metadata;
+using DragoAnt.System.Text.Json.Observer.Strategies;
+using static DragoAnt.System.Text.Json.Observer.JsonObserverValuePolicies;
+
+namespace DragoAnt.System.Text.Json.Observer.Tests.Shared;
+
+public abstract class CaseSensitivityTests
+{
+    private static readonly JsonObserverOptions Exact = new(PropertyNameCaseInsensitive: false);
+
+    private static readonly JsonObserver Rules = JsonObserver.Obj(Relative(b => b
+            .Match("password").MaskAny("1")
+            .Match(PropMatches.StartsWith("tok")).MaskAny("2")
+            .Match(PropMatches.EndsWith("Card")).MaskAny("3")
+            .Match(PropMatches.Contains("mail")).MaskAny("4")
+            .Match(PropMatches.OneOf("pin", "cvv")).MaskAny("5")
+            .Match("ключ").MaskAny("6")
+            .Match(PropMatches.StartsWith("пар")).MaskAny("7"),
+        BlockList));
+
+    private const string Payload = """{"password":"a","Password":"b","token":"c","Token":"d","myCard":"e","mycard":"f","email":"g","eMail":"h","pin":"i","PIN":"j","ключ":"k","КЛЮЧ":"l","пароль":"m","Пароль":"n"}""";
+
+    [Fact]
+    public void Rules_Default_IgnoreCase() =>
+        Rules.Mask(Payload).Should().Be(
+            """{"password":"1","Password":"1","token":"2","Token":"2","myCard":"3","mycard":"3","email":"4","eMail":"4","pin":"5","PIN":"5","ключ":"6","КЛЮЧ":"6","пароль":"7","Пароль":"7"}""");
+
+    [Fact]
+    public void Rules_CaseSensitive_MatchExactNamesOnly() =>
+        Rules.Mask(Payload, Exact).Should().Be(
+            """{"password":"1","Password":"b","token":"2","Token":"d","myCard":"3","mycard":"f","email":"4","eMail":"h","pin":"5","PIN":"j","ключ":"6","КЛЮЧ":"l","пароль":"7","Пароль":"n"}""");
+
+    [Fact]
+    public void AbsoluteRules_CaseSensitive_UnderAllowList_MaskUnmatchedCase() =>
+        JsonObserver.Obj(root => root.Match("order").Obj(o => o.Match("id").Unmasked()))
+            .Mask("""{"order":{"id":1,"ID":2},"Order":{"id":3}}""", Exact)
+            .Should().Be("""{"order":{"id":1,"ID":"***"},"Order":{"id":"***"}}""");
+
+    [Fact]
+    public void CustomRule_SeesTheCallsMatchingMode()
+    {
+        var modes = new List<bool>();
+        var observer = JsonObserver.Obj(b => b.Match("a").MaskValue((ref Utf8JsonReader _, JsonWriter writer, JsonObserveringEmptyContext _, ref PropertyPath path) =>
+        {
+            modes.Add(path.PropertyNameCaseInsensitive);
+            writer.WriteNullValue();
+        }), BlockList);
+
+        observer.Mask("""{"a":1}""");
+        observer.Mask("""{"a":1}""", Exact);
+
+        modes.Should().Equal(true, false);
+    }
+
+    [Fact]
+    public void Shape_FollowsSerializerOptions()
+    {
+        var general = new JsonSerializerOptions { TypeInfoResolver = new DefaultJsonTypeInfoResolver() };
+        var shape = JsonShape.FromTypeInfo(general.GetTypeInfo(typeof(Person)), p => p.Name == "Secret" ? MaskTag.Full : null);
+        const string json = """{"Name":"a","name":"b","Secret":"c","secret":"d"}""";
+
+        JsonObserver.FromShape(shape).Mask(json).Should().Be("""{"Name":"a","name":"b","Secret":"***","secret":"***"}""");
+        JsonObserver.FromShape(shape, JsonShapeOptions.FromSerializerOptions(general)).Mask(json)
+            .Should().Be("""{"Name":"a","name":"***","Secret":"***","secret":"***"}""");
+        JsonObserver.FromShape(shape).Mask(json, Exact).Should().Be("""{"Name":"a","name":"***","Secret":"***","secret":"***"}""");
+        JsonObserver.FromShape(shape, new JsonShapeOptions(PropertyNameCaseInsensitive: true)).Mask(json, Exact)
+            .Should().Be("""{"Name":"a","name":"b","Secret":"***","secret":"***"}""");
+        JsonShapeOptions.FromSerializerOptions(new JsonSerializerOptions(JsonSerializerDefaults.Web)).PropertyNameCaseInsensitive.Should().BeTrue();
+    }
+
+    [Fact]
+    public void Shape_CaseSensitive_KeepsNamesDifferingInCase()
+    {
+        var shape = JsonShape.Object(("id", JsonShape.Scalar), ("ID", JsonShape.Masked(MaskTag.Full)), ("ключ", JsonShape.Scalar));
+
+        JsonObserver.FromShape(shape).Mask("""{"id":1,"ID":2}""").Should().Be("""{"id":"***","ID":"***"}""");
+        JsonObserver.FromShape(shape).Mask("""{"id":1,"ID":2,"ключ":3,"КЛЮЧ":4}""", Exact)
+            .Should().Be("""{"id":1,"ID":"***","ключ":3,"КЛЮЧ":"***"}""");
+        shape.FindMember("Id"u8)!.Name.Should().Be("ID");
+        shape.FindMember("Id"u8, propertyNameCaseInsensitive: false).Should().BeNull();
+        shape.FindMember("id"u8, propertyNameCaseInsensitive: false)!.Name.Should().Be("id");
+        shape.FindMember(Encoding.UTF8.GetBytes("КЛЮЧ"), propertyNameCaseInsensitive: false).Should().BeNull();
+        shape.FindMember(Encoding.UTF8.GetBytes("КЛЮЧ"))!.Name.Should().Be("ключ");
+    }
+
+    public sealed class Person
+    {
+        public string? Name { get; set; }
+        public string? Secret { get; set; }
+    }
+}

DragoAnt.System.Text.Json.Observer.Tests.Shared/DefaultPolicyTests.cs

@@ -9,7 +9,7 @@ public abstract class DefaultPolicyTests
     [Fact]
     public void SharedNestedRule_TwoParentsDifferentDefaults_EachUsesOwn()
     {
-        var shared = JsonObserverItem<JsonObserveringEmptyContext>.Obj(b => b.Match("pin").MaskStr((_, _) => "***"), null);
+        var shared = JsonObserverItem<JsonObserveringEmptyContext>.Obj(b => b.Match("pin").MaskStr((_, _) => "***"), null).Delegate;
         var blockList = JsonObserver.Obj(b => b.Match("a").Obj(shared), BlockList);
         var nullList = JsonObserver.Obj(b => b.Match("a").Obj(shared), NullList);