diff --git a/CHANGELOG.md b/CHANGELOG.md index 48dd34e..a720b4c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,15 +33,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 2. **`Mask(string)` never throws.** It runs the same UTF-8 path as the bytes API and produces the same output for the same text: comments are skipped (never written), trailing commas are accepted, non-ASCII and HTML characters are written unescaped (`RelaxedEscaping`), invalid or cut-off text yields its masked prefix. `Mask(string, out MaskResult, options)` reports the status. The `JsonReaderOptions`, `JsonWriterOptions`, `ignoreNulls` and `ignoreComments` parameters are gone: use `JsonObserverOptions` (`IgnoreNulls`, `Indented`, `MaxDepth`). 3. **`Read(...)` never throws and returns a `MaskResult`** instead of `void`; `Read(byte[])` became `Read(ReadOnlySpan)`. 4. **Every `Mask*` rule masks the whole value whatever its JSON type.** `MaskStr`, `MaskRawValue`, `MaskInt`, `MaskLong`, `MaskDecimal` and `MaskBool` no longer pass a value of another type to the default policy (where `BlockList` exposed it), and no longer descend into an object or array: a container is skipped unread and the function receives `null`. `MaskStr` hands a number or boolean to its function as its literal (`"12.50"`, `"true"`). As these rules now match containers too, a rule written before an `Obj(...)` or `Array(...)` rule for the same name takes precedence over it. -5. **A string cut by `MaxValueBytes` reports `Truncated`** (with `FailedAtByte` `-1`), not `Masked`. A masking function receives a value longer than `MaxValueBytes` cut to that length. -6. **Number read rules no longer fail the body:** `ReadInt`, `ReadLong` and `ReadDecimal` receive `null` for a number that does not fit the type, and the token is written unchanged. +5. **A string cut by `MaxValueBytes` reports `Truncated`** (with `FailedAtByte` `-1`), not `Masked`. The cap applies to values written unmasked only (see 12). +6. **Number read rules no longer fail the body:** `ReadInt`, `ReadLong` and `ReadDecimal` receive `null` for a number that does not fit the type; the default policy writes the token (see 11). 7. **`PropertyPath` is a `ref struct` valid only during the call** it is passed to: `GetPropertyName`, `GetPropertyNameReverse`, `Length` and `ToString` remain; its constructor, `MaxLength` and `Dispose` are internal. Custom rules compiled against 1.x must be rebuilt. `ToString` writes array items as `[index]` (`a.b[0].c`, formerly `a.b..c`). 8. **`JsonWriter` can no longer be derived from outside the library**, `JsonWriter.FromUtf8JsonWriter` and `JsonWriter.Empty` are removed, and `WriteCommentValue` is gone (comments are never written). 9. **Internal now:** `JsonObserverException`, `PropertyPathMatch`, `JsonPropertyMatchDelegate`, `JsonPropertyPathMatchDelegate`, and the constructors of `JsonObjBuilder`, `JsonArrayBuilder`, `JsonValuePolicyBuilder` and their rule builders (start rules with `Match`). 10. **A UTF-8 byte order mark at the start of the input is skipped.** +11. **Read rules no longer decide what is written.** `ReadStr`, `ReadInt`, `ReadLong`, `ReadDecimal`, `ReadBool` and `ReadRaw` hand the value to the context; the next rule on the same match or the default policy writes it, so under `AllowList` a read value is `"***"`, not clear text. Chain `.Unmasked()` to keep it clear, or a mask method to read and mask one match: `Match("ssn").ReadStr(f).MaskAny(MaskTag.Last4)`. A read rule runs wherever it stands among the rules, so `Match("ssn").MaskAny(…).Match("ssn").ReadStr(f)` reads too. `Explain` reports `Read` only for a value read and written unchanged. +12. **A masking function receives the whole value, and mask output is never cut by `MaxValueBytes`.** A `Last4`-style function sees the real last characters (`***4444`, not the ones at the cap), and `MaskTag.Hash`, a function's result and the `AllowList` stars are written whole, with status `Masked`. A function therefore decodes a long value in full. ### Fixed +- **`PropMatches.Regex` follows `PropertyNameCaseInsensitive`:** under the default case-insensitive option it now also matches names that differ in case only (`DRiverLicensE` for `^driverLicense$`), as every other matcher does; a `Regex` built with `RegexOptions.IgnoreCase` ignores case under either option. A custom name test can follow the option through the new `PropMatchingStrategy(Func)` constructor. + - **Rules of an `Obj(...)` inside a property's `Array(...)` now apply** at any depth (`root.Match("lines").Array(l => l.Obj(…))`, and `Array(a => a.Array(b => b.Obj(…)))`). They looked for their names one or more levels too deep, so under `BlockList` those values were written in clear and under `AllowList` the whole item was masked. Only an `Obj(...)` directly under a root `Array(...)` worked. - **Relative rules receive `null` like absolute ones:** `MaskStr`, `MaskRawValue`, `MaskInt`, `MaskLong`, `MaskDecimal`, `MaskBool` and the read rules inside `Relative(...)` are called for a JSON `null`, as the rule-kinds table documents; `MaskAny` and `MaskAny(MaskTag)` keep `null` without calling the function. diff --git a/DragoAnt.System.Text.Json.Observer.Tests.Shared/ApiSurfaceTests.cs b/DragoAnt.System.Text.Json.Observer.Tests.Shared/ApiSurfaceTests.cs index 8b2bd9a..31060f1 100644 --- a/DragoAnt.System.Text.Json.Observer.Tests.Shared/ApiSurfaceTests.cs +++ b/DragoAnt.System.Text.Json.Observer.Tests.Shared/ApiSurfaceTests.cs @@ -40,12 +40,12 @@ public void RelativeBuilder_EveryReadRule_KeepsValues() { var context = new Values(); var observer = JsonObserver.Obj(JsonObserverValuePolicies.Relative(b => b - .Match("s").ReadStr((v, c) => c.Str = v) - .Match("i").ReadInt((v, c) => c.Int = v) - .Match("l").ReadLong((v, c) => c.Long = v) - .Match("d").ReadDecimal((v, c) => c.Decimal = v) - .Match("b").ReadBool((v, c) => c.Bool = v) - .Match("r").ReadRaw((v, c) => c.Raw = v) + .Match("s").ReadStr((v, c) => c.Str = v).Unmasked() + .Match("i").ReadInt((v, c) => c.Int = v).Unmasked() + .Match("l").ReadLong((v, c) => c.Long = v).Unmasked() + .Match("d").ReadDecimal((v, c) => c.Decimal = v).Unmasked() + .Match("b").ReadBool((v, c) => c.Bool = v).Unmasked() + .Match("r").ReadRaw((v, c) => c.Raw = v).Unmasked() .Match("m").MaskInt((_, _) => "i") .Match("n").MaskLong((_, _) => "l") .Match("o").MaskDecimal((_, _) => "d") diff --git a/DragoAnt.System.Text.Json.Observer.Tests.Shared/LeakTests.cs b/DragoAnt.System.Text.Json.Observer.Tests.Shared/LeakTests.cs new file mode 100644 index 0000000..94e008d --- /dev/null +++ b/DragoAnt.System.Text.Json.Observer.Tests.Shared/LeakTests.cs @@ -0,0 +1,270 @@ +using System.Buffers; +using System.Text; +using System.Text.RegularExpressions; +using DragoAnt.System.Text.Json.Observer.Strategies; + +namespace DragoAnt.System.Text.Json.Observer.Tests.Shared; + +public abstract class LeakTests +{ + private const string Ssn = """{"ssn":"123-45-6789","x":"y"}"""; + + private static JsonObserverValueDelegate AllowList => JsonObserverValuePolicies.AllowList; + private static JsonObserverValueDelegate BlockList => JsonObserverValuePolicies.BlockList; + private static JsonObserverValueDelegate NullList => JsonObserverValuePolicies.NullList; + + [Fact] + public void Read_UnderAllowList_WritesTheValueMasked() + { + var holder = new Holder(); + + JsonObserver.Obj(r => r.Match("ssn").ReadStr(Keep), AllowList).Mask(Ssn, holder) + .Should().Be("""{"ssn":"***","x":"***"}"""); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void Read_UnderBlockList_WritesTheValueUnchanged() + { + var holder = new Holder(); + + JsonObserver.Obj(r => r.Match("ssn").ReadStr(Keep), BlockList).Mask(Ssn, holder) + .Should().Be(Ssn); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void Read_UnderNullList_WritesNull() + { + var holder = new Holder(); + + JsonObserver.Obj(r => r.Match("ssn").ReadStr(Keep), NullList).Mask(Ssn, holder) + .Should().Be("""{"ssn":null,"x":null}"""); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void Read_InRelativePolicy_UnderAllowList_WritesTheValueMasked() + { + var holder = new Holder(); + var observer = JsonObserver.Obj(JsonObserverValuePolicies.Relative(b => b.Match("ssn").ReadStr(Keep))); + + observer.Mask("""{"person":{"ssn":"123-45-6789"}}""", holder).Should().Be("""{"person":{"ssn":"***"}}"""); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void ReadNumber_UnderAllowList_WritesTheValueMasked() + { + var holder = new Holder(); + + JsonObserver.Obj(r => r.Match("pin").ReadInt((v, h) => h.Number = v)).Mask("""{"pin":1234}""", holder) + .Should().Be("""{"pin":"***"}"""); + holder.Number.Should().Be(1234); + } + + [Fact] + public void Read_ThenUnmasked_WritesClearText() + { + var holder = new Holder(); + + JsonObserver.Obj(r => r.Match("ssn").ReadStr(Keep).Unmasked(), AllowList).Mask(Ssn, holder) + .Should().Be("""{"ssn":"123-45-6789","x":"***"}"""); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void Read_ThenMask_OnOneMatch_ReadsAndMasks() + { + var holder = new Holder(); + + JsonObserver.Obj(r => r.Match("ssn").ReadStr(Keep).MaskAny(MaskTag.Last4), BlockList).Mask(Ssn, holder) + .Should().Be("""{"ssn":"***6789","x":"y"}"""); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void Mask_ThenRead_OnOneMatch_ReadsAndMasks() + { + var holder = new Holder(); + + JsonObserver.Obj(r => r.Match("ssn").MaskAny(MaskTag.Full).Match("ssn").ReadStr(Keep), BlockList).Mask(Ssn, holder) + .Should().Be("""{"ssn":"***","x":"y"}"""); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void Read_ThenMask_InRelativePolicy_ReadsAndMasks() + { + var holder = new Holder(); + var observer = JsonObserver.Obj(JsonObserverValuePolicies.Relative( + b => b.Match("ssn").ReadStr(Keep).MaskAny(MaskTag.Full), BlockList)); + + observer.Mask("""{"person":{"ssn":"123-45-6789","x":"y"}}""", holder).Should().Be("""{"person":{"ssn":"***","x":"y"}}"""); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void ArrayRead_UnderAllowList_WritesTheItemMasked() + { + var holder = new Holder(); + + JsonObserver.Array(a => a.ReadStr(Keep)).Mask("""["123-45-6789"]""", holder).Should().Be("""["***"]"""); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void ArrayRead_ThenUnmaskedOrMask_DecidesTheItem() + { + var unmasked = new Holder(); + var masked = new Holder(); + + JsonObserver.Array(a => a.ReadStr(Keep).Unmasked()).Mask("""["123-45-6789"]""", unmasked).Should().Be("""["123-45-6789"]"""); + JsonObserver.Array(a => a.MaskAny(MaskTag.Last4).ReadStr(Keep), BlockList).Mask("""["123-45-6789"]""", masked) + .Should().Be("""["***6789"]"""); + unmasked.Value.Should().Be("123-45-6789"); + masked.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void Read_ExtractionOnly_StillReads() + { + var holder = new Holder(); + + JsonObserver.Obj(r => r.Match("ssn").MaskAny(MaskTag.Full).Match("ssn").ReadStr(Keep)).Read(Ssn, holder) + .Status.Should().Be(MaskStatus.Masked); + holder.Value.Should().Be("123-45-6789"); + } + + [Fact] + public void Explain_Read_ReportsWhatIsWritten() + { + JsonObserver.Obj(r => r.Match("ssn").ReadStr(Keep), AllowList).Explain("ssn").Outcome + .Should().Be(JsonPathOutcome.Masked); + JsonObserver.Obj(r => r.Match("ssn").ReadStr(Keep).Unmasked(), AllowList).Explain("ssn").Outcome + .Should().Be(JsonPathOutcome.Read); + JsonObserver.Obj(r => r.Match("ssn").ReadStr(Keep).MaskAny(MaskTag.Full), BlockList).Explain("ssn").Outcome + .Should().Be(JsonPathOutcome.Masked); + } + + public static TheoryData MaskFunctionKinds => ["MaskAny", "MaskStr", "MaskRawValue"]; + + [Theory] + [MemberData(nameof(MaskFunctionKinds))] + public void MaskFunction_UnderValueCap_ReceivesTheWholeValue(string kind) + { + var observer = JsonObserver.Obj(r => _ = kind switch + { + "MaskAny" => r.Match("card").MaskAny((s, _) => "***" + s![^4..]), + "MaskStr" => r.Match("card").MaskStr((s, _) => "***" + s![^4..]), + _ => r.Match("card").MaskRawValue((s, _) => "***" + s![^4..]), + }); + + observer.Mask("""{"card":"1111222233334444"}""", out var result, new JsonObserverOptions(MaxValueBytes: 8)) + .Should().Be("""{"card":"***4444"}"""); + result.Status.Should().Be(MaskStatus.Masked); + } + + [Fact] + public void MaskFunction_UnderValueCap_ReceivesTheWholeValue_FromSegments() + { + var observer = JsonObserver.Obj(r => r.Match("card").MaskAny((s, _) => "***" + s![^4..])); + var output = new ArrayBufferWriter(); + + var result = observer.Mask(SequenceInputTests.Split("""{"card":"1111222233334444"}"""u8.ToArray(), 3), output, + new JsonObserverOptions(MaxValueBytes: 8)); + + Encoding.UTF8.GetString(output.WrittenSpan).Should().Be("""{"card":"***4444"}"""); + result.Status.Should().Be(MaskStatus.Masked); + } + + [Fact] + public void MaskOutput_LongerThanValueCap_IsNotCut() + { + var observer = JsonObserver.Obj(r => r.Match("card").MaskAny((_, _) => "replaced-by-a-long-mask")); + + observer.Mask("""{"card":"1"}""", out var result, new JsonObserverOptions(MaxValueBytes: 8)) + .Should().Be("""{"card":"replaced-by-a-long-mask"}"""); + result.Status.Should().Be(MaskStatus.Masked); + } + + [Fact] + public void Hash_UnderValueCap_IsNotCut() + { + var observer = JsonObserver.Obj(r => r.Match("card").MaskAny(MaskTag.Hash)); + var options = new JsonObserverOptions(MaxValueBytes: 8, HashKey: "0123456789abcdef0123456789abcdef"u8.ToArray()); + + var masked = observer.Mask("""{"card":"1111222233334444"}""", out var result, options)!; + var uncapped = observer.Mask("""{"card":"1111222233334444"}""", options with { MaxValueBytes = int.MaxValue }); + + masked.Should().Be(uncapped); + JsonDocument.Parse(masked).RootElement.GetProperty("card").GetString().Should().StartWith("hash:").And.NotContain("…"); + result.Status.Should().Be(MaskStatus.Masked); + } + + [Fact] + public void UnmaskedValue_UnderValueCap_IsStillCut() + { + JsonObserver.Obj(JsonObserverValuePolicies.BlockList) + .Mask("""{"note":"1111222233334444"}""", out var result, new JsonObserverOptions(MaxValueBytes: 8)) + .Should().Be("""{"note":"11112222…"}"""); + result.Status.Should().Be(MaskStatus.Truncated); + } + + private const string CaseCorpus = + """{"driverLicense":"A1","DRiverLicensE":"A2","driverlicense":"A3","driverLicense":"A4","drіverLicense":"A5","drİverLicense":"A6"}"""; + + public static TheoryData CaseTruthTable => new() + { + { "Match", true, "A1 A2 A3 A4" }, + { "Match", false, "A1 A4" }, + { "Regex", true, "A1 A2 A3 A4" }, + { "Regex", false, "A1 A4" }, + { "Function", true, "A1 A2 A3 A4" }, + { "Function", false, "A1 A4" }, + { "RelativeRegex", true, "A1 A2 A3 A4" }, + { "RelativeRegex", false, "A1 A4" }, + }; + + [Theory] + [MemberData(nameof(CaseTruthTable))] + public void CaseOption_ReachesEveryMatcher(string matcher, bool caseInsensitive, string expectedMasked) + { + PropMatchingStrategy match = matcher switch + { + "Match" => "driverLicense", + "Function" => new PropMatchingStrategy((name, comparison) => string.Equals(name, "driverLicense", comparison)), + _ => PropMatches.Regex(new Regex("^driverLicense$")), + }; + var observer = matcher == "RelativeRegex" + ? JsonObserver.Obj(JsonObserverValuePolicies.Relative(b => b.Match(match).MaskAny(MaskTag.Full), JsonObserverValuePolicies.BlockList)) + : JsonObserver.Obj(r => r.Match(match).MaskAny(MaskTag.Full), JsonObserverValuePolicies.BlockList); + + var output = observer.Mask(CaseCorpus, new JsonObserverOptions(PropertyNameCaseInsensitive: caseInsensitive))!; + + var masked = JsonDocument.Parse(output).RootElement.EnumerateObject() + .Select((p, i) => (Key: $"A{i + 1}", Value: p.Value.GetString())) + .Where(p => p.Value == "***") + .Select(p => p.Key); + string.Join(' ', masked).Should().Be(expectedMasked); + } + + [Fact] + public void Regex_WithExplicitIgnoreCase_StaysCaseInsensitive() + { + var observer = JsonObserver.Obj( + r => r.Match(PropMatches.Regex(new Regex("^token$", RegexOptions.IgnoreCase))).MaskAny(MaskTag.Full), + JsonObserverValuePolicies.BlockList); + + observer.Mask("""{"Token":"a"}""", new JsonObserverOptions(PropertyNameCaseInsensitive: false)).Should().Be("""{"Token":"***"}"""); + } + + private static void Keep(string? value, Holder holder) => holder.Value = value; + + public sealed class Holder + { + public string? Value { get; set; } + + public int? Number { get; set; } + } +} diff --git a/DragoAnt.System.Text.Json.Observer.Tests.Shared/LimitsTests.cs b/DragoAnt.System.Text.Json.Observer.Tests.Shared/LimitsTests.cs index d71f368..3fa1a83 100644 --- a/DragoAnt.System.Text.Json.Observer.Tests.Shared/LimitsTests.cs +++ b/DragoAnt.System.Text.Json.Observer.Tests.Shared/LimitsTests.cs @@ -26,20 +26,22 @@ public void MaxValueBytes_NoValueCut_ReportsMasked() => BytesApiTests.Mask(Observer, """{"note":"abcde"}""", new JsonObserverOptions(MaxValueBytes: 5)).Result.Status.Should().Be(MaskStatus.Masked); [Fact] - public void MaxValueBytes_StrategyOutputCut_ReportsTruncated() + public void MaxValueBytes_StrategyOutput_IsNotCut() { var observer = JsonObserver.Obj(b => b.Match("a").MaskStr((_, _) => new string('x', 100)), BlockList); var (result, output) = BytesApiTests.Mask(observer, """{"a":"v"}""", new JsonObserverOptions(MaxValueBytes: 4)); - result.Status.Should().Be(MaskStatus.Truncated); - output.Should().Be("{\"a\":\"xxxx…\"}"); + result.Status.Should().Be(MaskStatus.Masked); + output.Should().Be($"{{\"a\":\"{new string('x', 100)}\"}}"); } [Fact] - public void MaxValueBytes_LongStrategyString_SurrogatePairNotSplit() + public void MaxValueBytes_LongCustomRuleString_SurrogatePairNotSplit() { - var observer = JsonObserver.Obj(b => b.Match("a").MaskStr((_, _) => "ab\U0001F600cd"), BlockList); + var observer = JsonObserver.Obj( + b => b.Match("a").MaskValue((ref Utf8JsonReader _, JsonWriter w, JsonObserveringEmptyContext _, ref PropertyPath _) => w.WriteStringValue("ab\U0001F600cd")), + BlockList); var (_, output) = BytesApiTests.Mask(observer, """{"a":"v"}""", new JsonObserverOptions(MaxValueBytes: 4)); @@ -47,23 +49,17 @@ public void MaxValueBytes_LongStrategyString_SurrogatePairNotSplit() } [Fact] - public void MaskAny_Strategy_HugeValue_DoesNotDecodeWhole() + public void MaskAny_Strategy_HugeValue_ReceivesTheWholeValue() { var secret = new string('s', 5 * 1024 * 1024); var utf8 = Encoding.UTF8.GetBytes($$"""{"password":"{{secret}}","n":1}"""); var observer = JsonObserver.Obj(Relative(b => b.Match("password").MaskAny((v, _) => v is null ? null : "len:" + v.Length), BlockList)); - var options = new JsonObserverOptions(MaxValueBytes: 256); var output = new ArrayBufferWriter(1024); - observer.Mask(utf8, output, options); - output.Clear(); - var before = GC.GetAllocatedBytesForCurrentThread(); - var result = observer.Mask(utf8, output, options); - var allocated = GC.GetAllocatedBytesForCurrentThread() - before; + var result = observer.Mask(utf8, output, new JsonObserverOptions(MaxValueBytes: 256)); result.Status.Should().Be(MaskStatus.Masked); - Encoding.UTF8.GetString(output.WrittenSpan).Should().Be("""{"password":"len:256","n":1}"""); - allocated.Should().BeLessThan(64 * 1024); + Encoding.UTF8.GetString(output.WrittenSpan).Should().Be($$"""{"password":"len:{{secret.Length}}","n":1}"""); } [Fact] diff --git a/DragoAnt.System.Text.Json.Observer.Tests/RunLeakTests.cs b/DragoAnt.System.Text.Json.Observer.Tests/RunLeakTests.cs new file mode 100644 index 0000000..4372603 --- /dev/null +++ b/DragoAnt.System.Text.Json.Observer.Tests/RunLeakTests.cs @@ -0,0 +1,3 @@ +namespace DragoAnt.System.Text.Json.Observer.Tests; + +public sealed class RunLeakTests : Shared.LeakTests; diff --git a/DragoAnt.System.Text.Json.Observer/BoundedJsonWriter.cs b/DragoAnt.System.Text.Json.Observer/BoundedJsonWriter.cs index 4e63b3f..af0005e 100644 --- a/DragoAnt.System.Text.Json.Observer/BoundedJsonWriter.cs +++ b/DragoAnt.System.Text.Json.Observer/BoundedJsonWriter.cs @@ -88,10 +88,15 @@ private void Start(JsonObserverOptions options) _safeDepth = 0; Exhausted = false; ValuesTruncated = false; + MaskOutput = false; } internal override bool Stopped => Exhausted; + internal override bool MaskOutput { get; set; } + + private int MaxValueBytes => MaskOutput ? int.MaxValue : _maxValueBytes; + private int Length => checked((int)(_writer.BytesCommitted + _writer.BytesPending)); public override void WriteNullValue() @@ -139,14 +144,14 @@ public override void WriteStringValue(ReadOnlySpan value) return; } - if ((long)value.Length * 3 <= _maxValueBytes) + if ((long)value.Length * 3 <= MaxValueBytes) { _writer.WriteStringValue(value); Completed(); return; } - var chars = value[..(int)Math.Min(value.Length, (long)_maxValueBytes + 1)]; + var chars = value[..(int)Math.Min(value.Length, (long)MaxValueBytes + 1)]; if (chars.Length < value.Length && char.IsHighSurrogate(chars[^1])) { chars = chars[..^1]; @@ -156,7 +161,7 @@ public override void WriteStringValue(ReadOnlySpan value) try { var utf8 = encoded.AsSpan(0, Encoding.UTF8.GetBytes(chars, encoded)); - if (chars.Length < value.Length && utf8.Length <= _maxValueBytes) + if (chars.Length < value.Length && utf8.Length <= MaxValueBytes) { WriteCut(utf8, utf8.Length); } @@ -178,14 +183,14 @@ public override void WriteStringValue(ReadOnlySpan utf8Value) return; } - if (utf8Value.Length <= _maxValueBytes) + if (utf8Value.Length <= MaxValueBytes) { _writer.WriteStringValue(utf8Value); Completed(); return; } - var cut = _maxValueBytes; + var cut = MaxValueBytes; while (cut > 0 && (utf8Value[cut] & 0xC0) == 0x80) { cut--; @@ -272,7 +277,7 @@ public override void WriteBase64StringValue(ReadOnlySpan bytes) } var length = Base64.GetMaxEncodedToUtf8Length(bytes.Length); - if (length <= _maxValueBytes) + if (length <= MaxValueBytes) { _writer.WriteBase64StringValue(bytes); Completed(); diff --git a/DragoAnt.System.Text.Json.Observer/Builders/JsonArrayBuilder.cs b/DragoAnt.System.Text.Json.Observer/Builders/JsonArrayBuilder.cs index e5f5d66..c12f71d 100644 --- a/DragoAnt.System.Text.Json.Observer/Builders/JsonArrayBuilder.cs +++ b/DragoAnt.System.Text.Json.Observer/Builders/JsonArrayBuilder.cs @@ -56,7 +56,7 @@ public JsonArrayBuilder MaskStr(StringMaskingStrategy strate /// public JsonArrayBuilder ReadStr(Action strategy) - => Read(JsonObserverItem.ReadStr(strategy, _builderDefaultValuePolicy), RuleText.ReadStr); + => Read(JsonObserverItem.ReadStr(strategy), RuleText.ReadStr); /// public JsonArrayBuilder MaskInt(Func strategy) @@ -64,7 +64,7 @@ public JsonArrayBuilder MaskInt(Func strategy /// public JsonArrayBuilder ReadInt(Action strategy) - => Read(JsonObserverItem.ReadInt(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadInt")); + => Read(JsonObserverItem.ReadInt(strategy), RuleText.ReadNumber("ReadInt")); /// public JsonArrayBuilder MaskLong(Func strategy) @@ -72,7 +72,7 @@ public JsonArrayBuilder MaskLong(Func strate /// public JsonArrayBuilder ReadLong(Action strategy) - => Read(JsonObserverItem.ReadLong(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadLong")); + => Read(JsonObserverItem.ReadLong(strategy), RuleText.ReadNumber("ReadLong")); /// public JsonArrayBuilder MaskDecimal(Func strategy) @@ -80,7 +80,7 @@ public JsonArrayBuilder MaskDecimal(Func /// public JsonArrayBuilder ReadDecimal(Action strategy) - => Read(JsonObserverItem.ReadDecimal(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadDecimal")); + => Read(JsonObserverItem.ReadDecimal(strategy), RuleText.ReadNumber("ReadDecimal")); /// public JsonArrayBuilder MaskBool(Func strategy) @@ -88,7 +88,7 @@ public JsonArrayBuilder MaskBool(Func strate /// public JsonArrayBuilder ReadBool(Action strategy) - => Read(JsonObserverItem.ReadBool(strategy, _builderDefaultValuePolicy), RuleText.ReadBool); + => Read(JsonObserverItem.ReadBool(strategy), RuleText.ReadBool); /// public JsonArrayBuilder MaskAny(Func strategy) @@ -108,7 +108,7 @@ public JsonArrayBuilder MaskRawValue(Func /// public JsonArrayBuilder ReadRaw(Action strategy) - => Read(JsonObserverItem.ReadRaw(strategy, _builderDefaultValuePolicy), RuleText.ReadRaw); + => Read(JsonObserverItem.ReadRaw(strategy), RuleText.ReadRaw); /// public JsonArrayBuilder MaskValue(JsonObserverValueDelegate policy) => @@ -135,8 +135,14 @@ internal JsonArrayBuilder MaskWhole(JsonObserverDelegate pol internal static (JsonObserverDelegate Delegate, RuleSet Set) Build(JsonArrayBuilder builder) => builder.Build(); - private JsonArrayBuilder Read(JsonObserverDelegate policy, string action) => - Add(type => type.IsValueToken(), policy, new RuleInfo(ValueItem, action, JsonPathOutcome.Read)); + private JsonArrayBuilder Read(JsonObserverItem.ReadValue read, string action) + { + _policies.Add(JsonObserverItem.Read( + (int _, ref PropertyPath _, JsonTokenType type) => (type.IsValueToken(), 1), + read, + new RuleInfo(ValueItem, action, JsonPathOutcome.Read))); + return this; + } private (JsonObserverDelegate, RuleSet) Build() { diff --git a/DragoAnt.System.Text.Json.Observer/Builders/JsonObjBuilder.cs b/DragoAnt.System.Text.Json.Observer/Builders/JsonObjBuilder.cs index 58dd4f4..3868fa4 100644 --- a/DragoAnt.System.Text.Json.Observer/Builders/JsonObjBuilder.cs +++ b/DragoAnt.System.Text.Json.Observer/Builders/JsonObjBuilder.cs @@ -44,7 +44,16 @@ private JsonObjBuilder AddAny(JsonPropertyPathMatchDelegate propNameMa Add((int depth, ref PropertyPath path, JsonTokenType _) => propNameMatch(depth, ref path), policy, info); private JsonObjBuilder AddValue(JsonPropertyPathMatchDelegate propNameMatch, JsonObserverDelegate policy, RuleInfo info) => - Add((int depth, ref PropertyPath path, JsonTokenType type) => + Add(ValueMatch(propNameMatch), policy, info); + + private JsonObjBuilder AddRead(JsonPropertyPathMatchDelegate propNameMatch, JsonObserverItem.ReadValue read, RuleInfo info) + { + _policies.Add(JsonObserverItem.Read(ValueMatch(propNameMatch), read, info)); + return this; + } + + private static JsonPropertyMatchDelegate ValueMatch(JsonPropertyPathMatchDelegate propNameMatch) => + (int depth, ref PropertyPath path, JsonTokenType type) => { var (success, propDepth) = propNameMatch(depth, ref path); @@ -54,7 +63,7 @@ private JsonObjBuilder AddValue(JsonPropertyPathMatchDelegate propName } return (true, propDepth); - }, policy, info); + }; private JsonObjBuilder Add(JsonPropertyMatchDelegate propMatch, JsonObserverDelegate policy, RuleInfo info) { @@ -90,40 +99,40 @@ public JsonObjBuilder MaskStr(StringMaskingStrategy strategy MaskWhole(JsonObserverItem.ApplyStringPolicy(strategy, strategy.Constant), RuleText.Strategy("MaskStr", strategy.Constant)); /// - public JsonObjBuilder ReadStr(Action strategy) - => Read(JsonObserverItem.ReadStr(strategy, _builderDefaultValuePolicy), RuleText.ReadStr); + public ReadRuleBuilder ReadStr(Action strategy) + => Read(JsonObserverItem.ReadStr(strategy), RuleText.ReadStr); /// public JsonObjBuilder MaskInt(Func strategy) => MaskWhole(JsonObserverItem.ApplyIntPolicy(strategy), "MaskInt(function)"); /// - public JsonObjBuilder ReadInt(Action strategy) - => Read(JsonObserverItem.ReadInt(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadInt")); + public ReadRuleBuilder ReadInt(Action strategy) + => Read(JsonObserverItem.ReadInt(strategy), RuleText.ReadNumber("ReadInt")); /// public JsonObjBuilder MaskLong(Func strategy) => MaskWhole(JsonObserverItem.ApplyLongPolicy(strategy), "MaskLong(function)"); /// - public JsonObjBuilder ReadLong(Action strategy) - => Read(JsonObserverItem.ReadLong(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadLong")); + public ReadRuleBuilder ReadLong(Action strategy) + => Read(JsonObserverItem.ReadLong(strategy), RuleText.ReadNumber("ReadLong")); /// public JsonObjBuilder MaskDecimal(Func strategy) => MaskWhole(JsonObserverItem.ApplyDecimalPolicy(strategy), "MaskDecimal(function)"); /// - public JsonObjBuilder ReadDecimal(Action strategy) - => Read(JsonObserverItem.ReadDecimal(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadDecimal")); + public ReadRuleBuilder ReadDecimal(Action strategy) + => Read(JsonObserverItem.ReadDecimal(strategy), RuleText.ReadNumber("ReadDecimal")); /// public JsonObjBuilder MaskBool(Func strategy) => MaskWhole(JsonObserverItem.ApplyBoolPolicy(strategy), "MaskBool(function)"); /// - public JsonObjBuilder ReadBool(Action strategy) - => Read(JsonObserverItem.ReadBool(strategy, _builderDefaultValuePolicy), RuleText.ReadBool); + public ReadRuleBuilder ReadBool(Action strategy) + => Read(JsonObserverItem.ReadBool(strategy), RuleText.ReadBool); /// public JsonObjBuilder MaskAny(Func strategy) @@ -142,8 +151,8 @@ public JsonObjBuilder MaskRawValue(Func st => MaskWhole(JsonObserverItem.ApplyRawPolicy(strategy), "MaskRawValue(function)"); /// - public JsonObjBuilder ReadRaw(Action strategy) - => Read(JsonObserverItem.ReadRaw(strategy, _builderDefaultValuePolicy), RuleText.ReadRaw); + public ReadRuleBuilder ReadRaw(Action strategy) + => Read(JsonObserverItem.ReadRaw(strategy), RuleText.ReadRaw); /// public JsonObjBuilder MaskValue(JsonObserverValueDelegate policy) => @@ -208,8 +217,8 @@ public JsonObjBuilder Array(JsonObserverDelegate policy) => internal JsonObjBuilder MaskWhole(JsonObserverDelegate policy, string action) => _builder.AddAny(_propNameMatch.AbsoluteMatch, policy, Info(action, JsonPathOutcome.Masked)); - private JsonObjBuilder Read(JsonObserverDelegate policy, string action) - => _builder.AddValue(_propNameMatch.AbsoluteMatch, policy, Info(action, JsonPathOutcome.Read)); + private ReadRuleBuilder Read(JsonObserverItem.ReadValue read, string action) + => new(this, _builder.AddRead(_propNameMatch.AbsoluteMatch, read, Info(action, JsonPathOutcome.Read))); private JsonObjBuilder Container(JsonTokenType container, JsonObserverDelegate policy, RuleInfo info) { @@ -230,4 +239,74 @@ private JsonObjBuilder Container(JsonTokenType container, JsonObserver private RuleInfo Info(string action, JsonPathOutcome outcome, RuleSet? child = null) => new(_propNameMatch.Describe(), action, outcome, child); } + + /// + /// A read rule just added. A read rule does not decide what is written: the default policy writes the value unless + /// or a mask method is chained here, which applies to the same match. + /// starts the next rule. + /// + public readonly ref struct ReadRuleBuilder + { + private readonly PropertyMaskingStrategyBuilder _rule; + private readonly JsonObjBuilder _builder; + + internal ReadRuleBuilder(PropertyMaskingStrategyBuilder rule, JsonObjBuilder builder) + { + _rule = rule; + _builder = builder; + } + + /// + public PropertyMaskingStrategyBuilder Match(PropMatchingStrategy match) => _builder.Match(match); + + /// + public PropertyMaskingStrategyBuilder Match(params PropMatchingStrategy[] match) => _builder.Match(match); + + /// + /// Writes the read value unchanged instead of through the default policy. + /// + public JsonObjBuilder Unmasked() => _rule.Unmasked(); + + /// + public JsonObjBuilder MaskStr(Func strategy) => _rule.MaskStr(strategy); + + /// + public JsonObjBuilder MaskStr(StringMaskingStrategy strategy) => _rule.MaskStr(strategy); + + /// + public JsonObjBuilder MaskInt(Func strategy) => _rule.MaskInt(strategy); + + /// + public JsonObjBuilder MaskLong(Func strategy) => _rule.MaskLong(strategy); + + /// + public JsonObjBuilder MaskDecimal(Func strategy) => _rule.MaskDecimal(strategy); + + /// + public JsonObjBuilder MaskBool(Func strategy) => _rule.MaskBool(strategy); + + /// + public JsonObjBuilder MaskAny(Func strategy) => _rule.MaskAny(strategy); + + /// + public JsonObjBuilder MaskAny(StringMaskingStrategy strategy) => _rule.MaskAny(strategy); + + /// + public JsonObjBuilder MaskAny(MaskTag tag) => _rule.MaskAny(tag); + + /// + public JsonObjBuilder MaskRawValue(Func strategy) => _rule.MaskRawValue(strategy); + + /// + public JsonObjBuilder MaskValue(JsonObserverValueDelegate policy) => _rule.MaskValue(policy); + + /// + public JsonObjBuilder MaskValue(JsonObserverDelegate policy) => _rule.MaskValue(policy); + + /// + /// The rules added so far, to keep adding to them. + /// + /// The read rule just added. + public static implicit operator JsonObjBuilder(ReadRuleBuilder rule) => rule._builder; + } } diff --git a/DragoAnt.System.Text.Json.Observer/Builders/JsonValuePolicyBuilder.cs b/DragoAnt.System.Text.Json.Observer/Builders/JsonValuePolicyBuilder.cs index 5977a37..35687c5 100644 --- a/DragoAnt.System.Text.Json.Observer/Builders/JsonValuePolicyBuilder.cs +++ b/DragoAnt.System.Text.Json.Observer/Builders/JsonValuePolicyBuilder.cs @@ -41,7 +41,18 @@ private JsonValuePolicyBuilder AddAnyProp(JsonPropertyPathMatchDelegat private JsonValuePolicyBuilder AddValueProp(JsonPropertyPathMatchDelegate propNameMatch, JsonObserverDelegate policy, RuleInfo info) { - var item = new JsonObserverItem((int depth, ref PropertyPath path, JsonTokenType type) => + _policies.Add(new JsonObserverItem(ValueMatch(propNameMatch), policy) { Info = info }); + return this; + } + + private JsonValuePolicyBuilder AddReadProp(JsonPropertyPathMatchDelegate propNameMatch, JsonObserverItem.ReadValue read, RuleInfo info) + { + _policies.Add(JsonObserverItem.Read(ValueMatch(propNameMatch), read, info)); + return this; + } + + private static JsonPropertyMatchDelegate ValueMatch(JsonPropertyPathMatchDelegate propNameMatch) => + (int depth, ref PropertyPath path, JsonTokenType type) => { var (success, nextDepth) = propNameMatch(depth, ref path); @@ -51,10 +62,7 @@ private JsonValuePolicyBuilder AddValueProp(JsonPropertyPathMatchDeleg } return (true, nextDepth); - }, policy) { Info = info }; - _policies.Add(item); - return this; - } + }; private JsonObserverDelegate Build() => JsonObserverItem.ApplyValuePolicy([.. _policies], _builderDefaultValuePolicy); @@ -80,8 +88,8 @@ internal PropertyMaskingStrategyBuilder( /// /// Masks the whole value with , whatever its JSON type: a string arrives decoded, /// a number or boolean as its JSON literal ("12.50", "true"), null as null, and an object - /// or array is skipped unread and arrives as null. A value longer than - /// arrives cut to that length. + /// or array is skipped unread and arrives as null. The strategy receives the whole value and its result is not cut by + /// . /// /// Returns the replacement string; null writes null. public JsonValuePolicyBuilder MaskStr(Func strategy) @@ -90,8 +98,8 @@ public JsonValuePolicyBuilder MaskStr(Func /// /// Masks the whole value with , whatever its JSON type: a string arrives decoded, /// a number or boolean as its JSON literal ("12.50", "true"), null as null, and an object - /// or array is skipped unread and arrives as null. A value longer than - /// arrives cut to that length. + /// or array is skipped unread and arrives as null. The strategy receives the whole value and its result is not cut by + /// . /// /// /// Replacement: a constant string, a whose matches become *, @@ -101,12 +109,13 @@ public JsonValuePolicyBuilder MaskStr(StringMaskingStrategy => MaskWhole(JsonObserverItem.ApplyStringPolicy(strategy, strategy.Constant), RuleText.Strategy("MaskStr", strategy.Constant)); /// - /// Hands a string or null value to and writes it unchanged. - /// A value of another type is not read and gets the default policy. + /// Hands a string or null value to ; a value of another type is not read. + /// Reading does not decide the output: the default policy writes the value unless + /// or a mask method is chained, or another rule writes the same match. /// /// Receives the decoded value and the context. - public JsonValuePolicyBuilder ReadStr(Action strategy) - => Read(JsonObserverItem.ReadStr(strategy, _builderDefaultValuePolicy), RuleText.ReadStr); + public ReadRuleBuilder ReadStr(Action strategy) + => Read(JsonObserverItem.ReadStr(strategy), RuleText.ReadStr); /// /// Masks the whole value with , whatever its JSON type. The strategy receives the number @@ -118,12 +127,13 @@ public JsonValuePolicyBuilder MaskInt(Func st => MaskWhole(JsonObserverItem.ApplyIntPolicy(strategy), "MaskInt(function)"); /// - /// Hands a number or null value to and writes it unchanged; a number that does not - /// fit arrives as null. A value of another type is not read and gets the default policy. + /// Hands a number or null value to ; a number that does not fit + /// arrives as null, and a value of another type is not read. Reading does not decide the output: the default policy writes the value unless + /// or a mask method is chained, or another rule writes the same match. /// /// Receives the value and the context. - public JsonValuePolicyBuilder ReadInt(Action strategy) - => Read(JsonObserverItem.ReadInt(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadInt")); + public ReadRuleBuilder ReadInt(Action strategy) + => Read(JsonObserverItem.ReadInt(strategy), RuleText.ReadNumber("ReadInt")); /// /// Masks the whole value with , whatever its JSON type. The strategy receives the number @@ -135,12 +145,13 @@ public JsonValuePolicyBuilder MaskLong(Func => MaskWhole(JsonObserverItem.ApplyLongPolicy(strategy), "MaskLong(function)"); /// - /// Hands a number or null value to and writes it unchanged; a number that does not - /// fit arrives as null. A value of another type is not read and gets the default policy. + /// Hands a number or null value to ; a number that does not fit + /// arrives as null, and a value of another type is not read. Reading does not decide the output: the default policy writes the value unless + /// or a mask method is chained, or another rule writes the same match. /// /// Receives the value and the context. - public JsonValuePolicyBuilder ReadLong(Action strategy) - => Read(JsonObserverItem.ReadLong(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadLong")); + public ReadRuleBuilder ReadLong(Action strategy) + => Read(JsonObserverItem.ReadLong(strategy), RuleText.ReadNumber("ReadLong")); /// /// Masks the whole value with , whatever its JSON type. The strategy receives the number @@ -152,12 +163,13 @@ public JsonValuePolicyBuilder MaskDecimal(Func MaskWhole(JsonObserverItem.ApplyDecimalPolicy(strategy), "MaskDecimal(function)"); /// - /// Hands a number or null value to and writes it unchanged; a number out of the - /// range arrives as null. A value of another type is not read and gets the default policy. + /// Hands a number or null value to ; a number out of the range + /// arrives as null, and a value of another type is not read. Reading does not decide the output: the default policy writes the value unless + /// or a mask method is chained, or another rule writes the same match. /// /// Receives the value, parsed with the invariant culture, and the context. - public JsonValuePolicyBuilder ReadDecimal(Action strategy) - => Read(JsonObserverItem.ReadDecimal(strategy, _builderDefaultValuePolicy), RuleText.ReadNumber("ReadDecimal")); + public ReadRuleBuilder ReadDecimal(Action strategy) + => Read(JsonObserverItem.ReadDecimal(strategy), RuleText.ReadNumber("ReadDecimal")); /// /// Masks the whole value with , whatever its JSON type. The strategy receives @@ -168,18 +180,19 @@ public JsonValuePolicyBuilder MaskBool(Func => MaskWhole(JsonObserverItem.ApplyBoolPolicy(strategy), "MaskBool(function)"); /// - /// Hands a boolean or null value to and writes it unchanged. - /// A value of another type is not read and gets the default policy. + /// Hands a boolean or null value to ; a value of another type is not read. + /// Reading does not decide the output: the default policy writes the value unless + /// or a mask method is chained, or another rule writes the same match. /// /// Receives the value and the context. - public JsonValuePolicyBuilder ReadBool(Action strategy) - => Read(JsonObserverItem.ReadBool(strategy, _builderDefaultValuePolicy), RuleText.ReadBool); + public ReadRuleBuilder ReadBool(Action strategy) + => Read(JsonObserverItem.ReadBool(strategy), RuleText.ReadBool); /// /// Masks the whole value with , whatever its JSON type: a string arrives decoded, /// a number or boolean as its JSON literal, and an object or array is skipped unread and arrives as null. - /// A null value stays null without calling the strategy. A value longer than - /// arrives cut to that length. + /// A null value stays null without calling the strategy. The strategy receives the whole value and its result is not cut by + /// . /// /// Returns the replacement string; null writes null. public JsonValuePolicyBuilder MaskAny(Func strategy) @@ -188,8 +201,8 @@ public JsonValuePolicyBuilder MaskAny(Func /// /// Masks the whole value with , whatever its JSON type: a string arrives decoded, /// a number or boolean as its JSON literal, and an object or array is skipped unread and arrives as null. - /// A null value stays null without calling the strategy. A value longer than - /// arrives cut to that length. + /// A null value stays null without calling the strategy. The strategy receives the whole value and its result is not cut by + /// . /// /// /// Replacement: a constant string, a whose matches become *, @@ -215,12 +228,13 @@ public JsonValuePolicyBuilder MaskRawValue(Func MaskWhole(JsonObserverItem.ApplyRawPolicy(strategy), "MaskRawValue(function)"); /// - /// Hands a string, number, boolean or null value to as its raw JSON text and writes - /// it unchanged. An object or array is not read and gets the default policy. + /// Hands a string, number, boolean or null value to as its raw JSON text; an object + /// or array is not read. Reading does not decide the output: the default policy writes the value unless + /// or a mask method is chained, or another rule writes the same match. /// /// Receives the raw text (a string without quotes, escapes kept) and the context. - public JsonValuePolicyBuilder ReadRaw(Action strategy) - => Read(JsonObserverItem.ReadRaw(strategy, _builderDefaultValuePolicy), RuleText.ReadRaw); + public ReadRuleBuilder ReadRaw(Action strategy) + => Read(JsonObserverItem.ReadRaw(strategy), RuleText.ReadRaw); /// /// Writes the value of the matched property unchanged. Applies to strings, numbers, booleans and null; @@ -262,8 +276,8 @@ public JsonValuePolicyBuilder MaskValue(JsonObserverDelegate internal JsonValuePolicyBuilder MaskWhole(JsonObserverDelegate policy, string action) => _builder.AddAnyProp(Match, policy, Info(action, JsonPathOutcome.Masked)); - private JsonValuePolicyBuilder Read(JsonObserverDelegate policy, string action) - => Value(policy, action, JsonPathOutcome.Read); + private ReadRuleBuilder Read(JsonObserverItem.ReadValue read, string action) + => new(this, _builder.AddReadProp(Match, read, Info(action, JsonPathOutcome.Read))); private JsonValuePolicyBuilder Value(JsonObserverDelegate policy, string action, JsonPathOutcome outcome) => _builder.AddValueProp(Match, policy, Info(action, outcome)); @@ -272,4 +286,71 @@ private JsonValuePolicyBuilder Value(JsonObserverDelegate po private RuleInfo Info(string action, JsonPathOutcome outcome) => new(_propNameMatch.Describe(), action, outcome); } + + /// + /// A read rule just added. A read rule does not decide what is written: the default policy writes the value unless + /// or a mask method is chained here, which applies to the same match. + /// starts the next rule. + /// + public readonly ref struct ReadRuleBuilder + { + private readonly PropertyMaskingStrategyBuilder _rule; + private readonly JsonValuePolicyBuilder _builder; + + internal ReadRuleBuilder(PropertyMaskingStrategyBuilder rule, JsonValuePolicyBuilder builder) + { + _rule = rule; + _builder = builder; + } + + /// + public PropertyMaskingStrategyBuilder Match(params PropMatchingStrategy[] match) => _builder.Match(match); + + /// + /// Writes the read value unchanged instead of through the default policy. + /// + public JsonValuePolicyBuilder Unmasked() => _rule.Unmasked(); + + /// + public JsonValuePolicyBuilder MaskStr(Func strategy) => _rule.MaskStr(strategy); + + /// + public JsonValuePolicyBuilder MaskStr(StringMaskingStrategy strategy) => _rule.MaskStr(strategy); + + /// + public JsonValuePolicyBuilder MaskInt(Func strategy) => _rule.MaskInt(strategy); + + /// + public JsonValuePolicyBuilder MaskLong(Func strategy) => _rule.MaskLong(strategy); + + /// + public JsonValuePolicyBuilder MaskDecimal(Func strategy) => _rule.MaskDecimal(strategy); + + /// + public JsonValuePolicyBuilder MaskBool(Func strategy) => _rule.MaskBool(strategy); + + /// + public JsonValuePolicyBuilder MaskAny(Func strategy) => _rule.MaskAny(strategy); + + /// + public JsonValuePolicyBuilder MaskAny(StringMaskingStrategy strategy) => _rule.MaskAny(strategy); + + /// + public JsonValuePolicyBuilder MaskAny(MaskTag tag) => _rule.MaskAny(tag); + + /// + public JsonValuePolicyBuilder MaskRawValue(Func strategy) => _rule.MaskRawValue(strategy); + + /// + public JsonValuePolicyBuilder MaskValue(JsonObserverValueDelegate policy) => _rule.MaskValue(policy); + + /// + public JsonValuePolicyBuilder MaskValue(JsonObserverDelegate policy) => _rule.MaskValue(policy); + + /// + /// The rules added so far, to keep adding to them. + /// + /// The read rule just added. + public static implicit operator JsonValuePolicyBuilder(ReadRuleBuilder rule) => rule._builder; + } } diff --git a/DragoAnt.System.Text.Json.Observer/Builders/RuleText.cs b/DragoAnt.System.Text.Json.Observer/Builders/RuleText.cs index 68134aa..d2c6aeb 100644 --- a/DragoAnt.System.Text.Json.Observer/Builders/RuleText.cs +++ b/DragoAnt.System.Text.Json.Observer/Builders/RuleText.cs @@ -9,12 +9,11 @@ internal static class RuleText { public const string Unmasked = "Unmasked()"; public const string CustomValue = "MaskValue(custom rule)"; - public const string ReadStr = "ReadStr (a string or null is read and written as is; another type gets the default policy)"; - public const string ReadBool = "ReadBool (a boolean or null is read and written as is; another type gets the default policy)"; - public const string ReadRaw = "ReadRaw (a scalar is read and written as is; an object or array gets the default policy)"; + public const string ReadStr = "ReadStr (a string or null is read)"; + public const string ReadBool = "ReadBool (a boolean or null is read)"; + public const string ReadRaw = "ReadRaw (a scalar is read)"; - public static string ReadNumber(string method) => - $"{method} (a number or null is read and written as is; another type gets the default policy)"; + public static string ReadNumber(string method) => $"{method} (a number or null is read)"; public static string Strategy(string method, string? constant) => constant is null ? $"{method}(function)" : $"{method}(\"{constant}\")"; diff --git a/DragoAnt.System.Text.Json.Observer/JsonObserverItem.cs b/DragoAnt.System.Text.Json.Observer/JsonObserverItem.cs index 980860b..015b7a2 100644 --- a/DragoAnt.System.Text.Json.Observer/JsonObserverItem.cs +++ b/DragoAnt.System.Text.Json.Observer/JsonObserverItem.cs @@ -94,87 +94,81 @@ public static (JsonObserverDelegate Delegate, RuleSet Set) A return JsonArrayBuilder.Build(builder); } - public static JsonObserverDelegate ReadStr(Action read, JsonObserverValueDelegate? valuePolicy) => - ApplyReadPolicy( - (ref Utf8JsonReader reader, TContext context) => read(reader.TokenType is Null ? null : reader.GetString(), context), - static type => type is JsonTokenType.String or Null, - valuePolicy); - - public static JsonObserverDelegate ReadInt(Action read, JsonObserverValueDelegate? valuePolicy) => - ApplyReadPolicy( - (ref Utf8JsonReader reader, TContext context) => read(reader.TokenType is Number && reader.TryGetInt32(out var v) ? v : null, context), - static type => type is Number or Null, - valuePolicy); - - public static JsonObserverDelegate ReadLong(Action read, JsonObserverValueDelegate? valuePolicy) => - ApplyReadPolicy( - (ref Utf8JsonReader reader, TContext context) => read(reader.TokenType is Number && reader.TryGetInt64(out var v) ? v : null, context), - static type => type is Number or Null, - valuePolicy); - - public static JsonObserverDelegate ReadDecimal(Action read, JsonObserverValueDelegate? valuePolicy) => - ApplyReadPolicy( - (ref Utf8JsonReader reader, TContext context) => read(reader.TokenType is Number && reader.TryGetDecimal(out var v) ? v : null, context), - static type => type is Number or Null, - valuePolicy); - - public static JsonObserverDelegate ReadBool(Action read, JsonObserverValueDelegate? valuePolicy) => - ApplyReadPolicy( - (ref Utf8JsonReader reader, TContext context) => read(reader.TokenType is Null ? null : reader.GetBoolean(), context), - static type => type is True or False or Null, - valuePolicy); - - public static JsonObserverDelegate ReadRaw(Action read, JsonObserverValueDelegate? valuePolicy) => - ApplyReadPolicy( - (ref Utf8JsonReader reader, TContext context) => - read(reader.HasValueSequence ? Encoding.UTF8.GetString(reader.ValueSequence) : Encoding.UTF8.GetString(reader.ValueSpan), context), - static type => type is JsonTokenType.String or Number or True or False or Null, - valuePolicy); - - private delegate void ReadValue(ref Utf8JsonReader reader, TContext context); - /// - /// Hands the value to and writes the token back unchanged. + /// What the rule reads; null for a rule that writes the value. /// - private static JsonObserverDelegate ApplyReadPolicy( - ReadValue read, - Func accepts, - JsonObserverValueDelegate? valuePolicy) - { - return ( - ref Utf8JsonReader reader, - JsonWriter writer, - TContext context, - int _, - ref PropertyPath propPath, - JsonObserverValueDelegate defaultValuePolicy) => + public ReadValue? Reader { get; init; } + + public static ReadValue ReadStr(Action read) => + (ref Utf8JsonReader reader, TContext context) => { - var tokenType = reader.TokenType; - if (!accepts(tokenType)) + if (reader.TokenType is JsonTokenType.String or Null) { - (valuePolicy ?? defaultValuePolicy).Invoke(ref reader, writer, context, ref propPath); - return; + read(reader.TokenType is Null ? null : reader.GetString(), context); } + }; - read(ref reader, context); - switch (tokenType) + public static ReadValue ReadInt(Action read) => + (ref Utf8JsonReader reader, TContext context) => + { + if (reader.TokenType is Number or Null) { - case Null: - writer.WriteNullValue(); - break; - case JsonTokenType.String: - writer.CopyStringValue(ref reader); - break; - case True: - case False: - writer.WriteBooleanValue(tokenType is True); - break; - default: - writer.CopyRawValue(ref reader); - break; + read(reader.TokenType is Number && reader.TryGetInt32(out var v) ? v : null, context); } }; - } + + public static ReadValue ReadLong(Action read) => + (ref Utf8JsonReader reader, TContext context) => + { + if (reader.TokenType is Number or Null) + { + read(reader.TokenType is Number && reader.TryGetInt64(out var v) ? v : null, context); + } + }; + + public static ReadValue ReadDecimal(Action read) => + (ref Utf8JsonReader reader, TContext context) => + { + if (reader.TokenType is Number or Null) + { + read(reader.TokenType is Number && reader.TryGetDecimal(out var v) ? v : null, context); + } + }; + + public static ReadValue ReadBool(Action read) => + (ref Utf8JsonReader reader, TContext context) => + { + if (reader.TokenType is True or False or Null) + { + read(reader.TokenType is Null ? null : reader.GetBoolean(), context); + } + }; + + public static ReadValue ReadRaw(Action read) => + (ref Utf8JsonReader reader, TContext context) => + { + if (reader.TokenType is JsonTokenType.String or Number or True or False or Null) + { + read(reader.HasValueSequence ? Encoding.UTF8.GetString(reader.ValueSequence) : Encoding.UTF8.GetString(reader.ValueSpan), context); + } + }; + + internal delegate void ReadValue(ref Utf8JsonReader reader, TContext context); + + /// + /// A read rule: hands the value to and leaves the writing to the next matching rule or the default policy. + /// + public static JsonObserverItem Read(JsonPropertyMatchDelegate propMatch, ReadValue read, RuleInfo info) => + new(propMatch, WriteByDefault) { Reader = read, Info = info }; + + private static void WriteByDefault( + ref Utf8JsonReader reader, + JsonWriter writer, + TContext context, + int _, + ref PropertyPath propPath, + JsonObserverValueDelegate defaultValuePolicy) => + defaultValuePolicy(ref reader, writer, context, ref propPath); /// /// Masks a value of any JSON type with the call's ; a container is skipped. @@ -207,8 +201,8 @@ public static JsonObserverDelegate ApplyTagPolicy(MaskTag tag) public static JsonObserverDelegate ApplyAnyPolicy(Func maskingRule, string? constant = null) => ApplyMaskPolicy( constant is not null - ? (ref Utf8JsonReader _, TContext _, int _) => constant - : (ref Utf8JsonReader reader, TContext context, int maxBytes) => maskingRule(ScalarText(ref reader, maxBytes, decode: true), context), + ? (ref Utf8JsonReader _, TContext _) => constant + : (ref Utf8JsonReader reader, TContext context) => maskingRule(ScalarText(ref reader, decode: true), context), keepNull: true); /// @@ -217,8 +211,8 @@ constant is not null public static JsonObserverDelegate ApplyStringPolicy(Func maskingRule, string? constant = null) => ApplyMaskPolicy( constant is not null - ? (ref Utf8JsonReader _, TContext _, int _) => constant - : (ref Utf8JsonReader reader, TContext context, int maxBytes) => maskingRule(ScalarText(ref reader, maxBytes, decode: true), context), + ? (ref Utf8JsonReader _, TContext _) => constant + : (ref Utf8JsonReader reader, TContext context) => maskingRule(ScalarText(ref reader, decode: true), context), keepNull: false); /// @@ -226,7 +220,7 @@ constant is not null /// public static JsonObserverDelegate ApplyRawPolicy(Func maskingRule) => ApplyMaskPolicy( - (ref Utf8JsonReader reader, TContext context, int maxBytes) => maskingRule(ScalarText(ref reader, maxBytes, decode: false), context), + (ref Utf8JsonReader reader, TContext context) => maskingRule(ScalarText(ref reader, decode: false), context), keepNull: false); /// @@ -234,7 +228,7 @@ public static JsonObserverDelegate ApplyRawPolicy(Func public static JsonObserverDelegate ApplyBoolPolicy(Func maskingRule) => ApplyMaskPolicy( - (ref Utf8JsonReader reader, TContext context, int _) => maskingRule(reader.TokenType switch + (ref Utf8JsonReader reader, TContext context) => maskingRule(reader.TokenType switch { True => true, False => false, @@ -247,7 +241,7 @@ public static JsonObserverDelegate ApplyBoolPolicy(Func public static JsonObserverDelegate ApplyIntPolicy(Func maskingRule) => ApplyMaskPolicy( - (ref Utf8JsonReader reader, TContext context, int _) => + (ref Utf8JsonReader reader, TContext context) => maskingRule(reader.TokenType is Number && reader.TryGetInt32(out var value) ? value : null, context), keepNull: false); @@ -256,7 +250,7 @@ public static JsonObserverDelegate ApplyIntPolicy(Func public static JsonObserverDelegate ApplyLongPolicy(Func maskingRule) => ApplyMaskPolicy( - (ref Utf8JsonReader reader, TContext context, int _) => + (ref Utf8JsonReader reader, TContext context) => maskingRule(reader.TokenType is Number && reader.TryGetInt64(out var value) ? value : null, context), keepNull: false); @@ -265,11 +259,11 @@ public static JsonObserverDelegate ApplyLongPolicy(Func public static JsonObserverDelegate ApplyDecimalPolicy(Func maskingRule) => ApplyMaskPolicy( - (ref Utf8JsonReader reader, TContext context, int _) => + (ref Utf8JsonReader reader, TContext context) => maskingRule(reader.TokenType is Number && reader.TryGetDecimal(out var value) ? value : null, context), keepNull: false); - private delegate string? MaskToken(ref Utf8JsonReader reader, TContext context, int maxValueBytes); + private delegate string? MaskToken(ref Utf8JsonReader reader, TContext context); /// /// Writes the function's replacement for the current value whatever its type, then moves past it; a container is never read. @@ -290,14 +284,16 @@ private static JsonObserverDelegate ApplyMaskPolicy(MaskToken mask, bo return; } - var result = mask(ref reader, context, writer.Options.MaxValueBytes); + var result = mask(ref reader, context); if (result is null) { writer.WriteNullValue(); } else { + writer.MaskOutput = true; writer.WriteStringValue(result); + writer.MaskOutput = false; } if (reader.TokenType is StartObject or StartArray && !reader.TrySkip()) @@ -308,9 +304,9 @@ private static JsonObserverDelegate ApplyMaskPolicy(MaskToken mask, bo } /// - /// Text of a string, number or boolean token, at most UTF-8 bytes of it; null for anything else. + /// The whole text of a string, number or boolean token; null for anything else. /// - private static string? ScalarText(ref Utf8JsonReader reader, int maxBytes, bool decode) + private static string? ScalarText(ref Utf8JsonReader reader, bool decode) { if (reader.TokenType is not (JsonTokenType.String or Number or True or False)) { @@ -323,7 +319,7 @@ private static JsonObserverDelegate ApplyMaskPolicy(MaskToken mask, bo var buffer = ArrayPool.Shared.Rent(length); try { - return Utf8Prefix(buffer.AsSpan(0, reader.CopyString(buffer)), maxBytes); + return Encoding.UTF8.GetString(buffer.AsSpan(0, reader.CopyString(buffer))); } finally { @@ -331,29 +327,7 @@ private static JsonObserverDelegate ApplyMaskPolicy(MaskToken mask, bo } } - if (!reader.HasValueSequence) - { - return Utf8Prefix(reader.ValueSpan, maxBytes); - } - - var sequence = reader.ValueSequence; - return Utf8Prefix(sequence.Slice(0, Math.Min(sequence.Length, (long)maxBytes + 4)).ToArray(), maxBytes); - } - - private static string Utf8Prefix(ReadOnlySpan utf8, int maxBytes) - { - if (utf8.Length > maxBytes) - { - var cut = Math.Max(maxBytes, 0); - while (cut > 0 && (utf8[cut] & 0xC0) == 0x80) - { - cut--; - } - - utf8 = utf8[..cut]; - } - - return Encoding.UTF8.GetString(utf8); + return reader.HasValueSequence ? Encoding.UTF8.GetString(reader.ValueSequence) : Encoding.UTF8.GetString(reader.ValueSpan); } /// @@ -364,6 +338,7 @@ private static string Utf8Prefix(ReadOnlySpan utf8, int maxBytes) public static JsonObserverDelegate ApplyValuePolicy(JsonObserverItem[] policies, JsonObserverValueDelegate? valuePolicy) { var defaultPolicy = GetApplyDefaultPolicy(valuePolicy, UnknownContainers.Create(policies)); + var lastReader = LastReader(policies); return ( ref Utf8JsonReader reader, @@ -381,6 +356,11 @@ public static JsonObserverDelegate ApplyValuePolicy(JsonObserverItem= 0) + { + RunReads(policies, lastReader, depth, ref propPath, ref reader, context); + } + var (matchPolicy, nextDepth) = MatchPolicy(policies, depth, ref propPath, tokenType); if (matchPolicy is not null) { @@ -415,6 +395,7 @@ private static JsonObserverDelegate ApplyObjPolicy( UnknownContainers unknown) { var defaultPolicy = GetApplyDefaultPolicy(valuePolicy, unknown); + var lastReader = LastReader(policies); return ( ref Utf8JsonReader reader, @@ -455,6 +436,10 @@ private static JsonObserverDelegate ApplyObjPolicy( } var tokenType = reader.TokenType; + if (lastReader >= 0) + { + RunReads(policies, lastReader, depth, ref propPath, ref reader, context); + } var (matchPolicy, nextDepth) = MatchPolicy(policies, depth, ref propPath, tokenType); @@ -496,6 +481,7 @@ private static JsonObserverDelegate ApplyArrayPolicy( UnknownContainers unknown) { var defaultPolicy = GetApplyDefaultPolicy(valuePolicy, unknown); + var lastReader = LastReader(policies); return ( ref Utf8JsonReader reader, @@ -535,6 +521,11 @@ private static JsonObserverDelegate ApplyArrayPolicy( var tokenType = reader.TokenType; propPath.AddArrayItem(index++); + if (lastReader >= 0) + { + RunReads(policies, lastReader, depth, ref propPath, ref reader, context); + } + var (matchPolicy, nextDepth) = MatchPolicy(policies, depth, ref propPath, tokenType); if (matchPolicy is not null) @@ -619,6 +610,11 @@ internal static (JsonObserverItem?, int depth) MatchPolicy( { foreach (var policyItem in policies) { + if (policyItem.Reader is not null) + { + continue; + } + var (success, nextDepth) = policyItem.Match(depth, ref path, tokenType); if (!success) { @@ -631,7 +627,35 @@ internal static (JsonObserverItem?, int depth) MatchPolicy( return (null, depth); } - private (bool success, int depth) Match(int depth, ref PropertyPath propPath, JsonTokenType token) => propMatch(depth, ref propPath, token); + /// + /// Index of the last read rule, or -1 when there is none, so that a pass without read rules skips . + /// + internal static int LastReader(JsonObserverItem[] policies) => + global::System.Array.FindLastIndex(policies, static p => p.Reader is not null); + + /// + /// Runs every read rule up to that matches the current value, wherever it stands + /// relative to the rule that writes the value. + /// + internal static void RunReads( + JsonObserverItem[] policies, + int lastReader, + int depth, + ref PropertyPath path, + ref Utf8JsonReader reader, + TContext context) + { + for (var i = 0; i <= lastReader; i++) + { + var policyItem = policies[i]; + if (policyItem.Reader is not null && policyItem.Match(depth, ref path, reader.TokenType).success) + { + policyItem.Reader(ref reader, context); + } + } + } + + internal (bool success, int depth) Match(int depth, ref PropertyPath propPath, JsonTokenType token) => propMatch(depth, ref propPath, token); internal void Apply( ref Utf8JsonReader reader, diff --git a/DragoAnt.System.Text.Json.Observer/JsonObserverOptions.cs b/DragoAnt.System.Text.Json.Observer/JsonObserverOptions.cs index c1a2664..63d76cb 100644 --- a/DragoAnt.System.Text.Json.Observer/JsonObserverOptions.cs +++ b/DragoAnt.System.Text.Json.Observer/JsonObserverOptions.cs @@ -8,7 +8,7 @@ namespace DragoAnt.System.Text.Json.Observer; /// Output size limit in UTF-8 bytes; when reached the output is closed and the status is . /// /// Longest string value written, in UTF-8 bytes; a longer one is cut, ends with an ellipsis and makes the status . -/// A rule's masking function also receives a longer value cut to this length. +/// It applies to values written unmasked: a masking function receives the whole value, and mask output, a hash included, is never cut. /// /// Deepest nesting accepted; a deeper payload is . /// Write non-ASCII and HTML-sensitive characters unescaped, which keeps logs readable. diff --git a/DragoAnt.System.Text.Json.Observer/JsonObserverValuePolicies.cs b/DragoAnt.System.Text.Json.Observer/JsonObserverValuePolicies.cs index c726c92..26d310a 100644 --- a/DragoAnt.System.Text.Json.Observer/JsonObserverValuePolicies.cs +++ b/DragoAnt.System.Text.Json.Observer/JsonObserverValuePolicies.cs @@ -158,7 +158,9 @@ public static void AllowList(ref Utf8JsonReader reader, JsonWriter writer, TCont case Number: case True: case False: + writer.MaskOutput = true; writer.WriteStringValue("***"u8); + writer.MaskOutput = false; break; case Null: writer.WriteNullValue(); @@ -177,10 +179,14 @@ public static void LegacyAllowList(ref Utf8JsonReader reader, JsonWriter writer, switch (reader.TokenType) { case JsonTokenType.String: + writer.MaskOutput = true; writer.WriteStringValue("#str#*****"); + writer.MaskOutput = false; break; case Number: + writer.MaskOutput = true; writer.WriteStringValue("#number#*****"); + writer.MaskOutput = false; break; case True: writer.WriteBooleanValue(true); diff --git a/DragoAnt.System.Text.Json.Observer/JsonWriter.cs b/DragoAnt.System.Text.Json.Observer/JsonWriter.cs index cfc4084..ddaf480 100644 --- a/DragoAnt.System.Text.Json.Observer/JsonWriter.cs +++ b/DragoAnt.System.Text.Json.Observer/JsonWriter.cs @@ -124,6 +124,15 @@ private protected JsonWriter() /// internal virtual bool Stopped => false; + /// + /// The strings written now replace a value, so does not cut them. + /// + internal virtual bool MaskOutput + { + get => false; + set { } + } + internal void CopyStringValue(ref Utf8JsonReader reader) { if (ReferenceEquals(this, Empty)) @@ -278,6 +287,12 @@ internal sealed class IgnoreNullsJsonWriter : JsonWriter, IDisposable internal override bool Stopped => _inner.Stopped; + internal override bool MaskOutput + { + get => _inner.MaskOutput; + set => _inner.MaskOutput = value; + } + public override void WriteNullValue() { if (_count > 0 && _pending[_count - 1].Kind == Kind.Name) diff --git a/DragoAnt.System.Text.Json.Observer/RuleExplainer.cs b/DragoAnt.System.Text.Json.Observer/RuleExplainer.cs index 04c5298..519266b 100644 --- a/DragoAnt.System.Text.Json.Observer/RuleExplainer.cs +++ b/DragoAnt.System.Text.Json.Observer/RuleExplainer.cs @@ -34,6 +34,12 @@ protected override (JsonPathOutcome Outcome, string Rule, string Action) Explain var token = TokenAt(segments, i, valueKind); var last = i == segments.Count - 1; var (item, nextDepth) = JsonObserverItem.MatchPolicy(items, depth, ref path, token); + var reads = last ? ReadsAt(items, depth, ref path, token) : []; + foreach (var read in reads) + { + steps.Add($"{at}: {read.Match} → {read.Action}"); + } + if (item is not null) { var info = item.Info; @@ -42,7 +48,7 @@ protected override (JsonPathOutcome Outcome, string Rule, string Action) Explain { var (outcome, action) = last ? Resolve(info, token) : (info.Outcome, $"{info.Action} on the whole {Container(token)}"); steps.Add($"{at}: {info.Match} → {action}"); - return (outcome, string.Join(" > ", chain), action); + return WithReads((outcome, string.Join(" > ", chain), action), chain.GetRange(0, chain.Count - 1), reads); } steps.Add($"{at}: {info.Match} → {info.Action}"); @@ -69,7 +75,7 @@ protected override (JsonPathOutcome Outcome, string Rule, string Action) Explain continue; } - return DefaultPolicy(effective, ref path, token, at, steps); + return WithReads(DefaultPolicy(effective, ref path, token, at, steps), chain, reads); } throw new InvalidOperationException("Unreachable: the last segment always returns."); @@ -89,16 +95,22 @@ private static (JsonPathOutcome, string, string) DefaultPolicy( { if (policy.Target is RelativeValuePolicy relative) { + var reads = ReadsAt(relative.Items, 0, ref path, token); + foreach (var read in reads) + { + steps.Add($"{at}: relative {read.Match} → {read.Action}"); + } + var (item, _) = JsonObserverItem.MatchPolicy(relative.Items, 0, ref path, token); if (item is not null) { var (relativeOutcome, relativeAction) = Resolve(item.Info, token); steps.Add($"{at}: relative {item.Info.Match} → {relativeAction}"); - return (relativeOutcome, $"relative {item.Info.Match}", relativeAction); + return WithReads((relativeOutcome, $"relative {item.Info.Match}", relativeAction), ["relative"], reads); } steps.Add($"{at}: no relative rule"); - return DefaultPolicy(relative.DefaultValuePolicy, ref path, token, at, steps); + return WithReads(DefaultPolicy(relative.DefaultValuePolicy, ref path, token, at, steps), ["relative"], reads); } var name = KnownPolicyName(policy); @@ -123,6 +135,39 @@ private static (JsonPathOutcome, string, string) DefaultPolicy( return (outcome, rule, action); } + private static List> ReadsAt(JsonObserverItem[] items, int depth, ref PropertyPath path, JsonTokenType token) + { + List> reads = []; + foreach (var item in items) + { + if (item.Reader is not null && item.Match(depth, ref path, token).success) + { + reads.Add(item.Info); + } + } + + return reads; + } + + /// + /// A value that read rules hand to the context is still written by the rule or policy that decided the result. + /// + private static (JsonPathOutcome, string, string) WithReads( + (JsonPathOutcome Outcome, string Rule, string Action) written, + List chain, + List> reads) + { + if (reads.Count == 0) + { + return written; + } + + var readRules = string.Join(" + ", reads.Select(r => string.Join(" > ", chain.Append(r.Match)))); + var readActions = string.Join("; ", reads.Select(r => r.Action)); + var outcome = written.Outcome is JsonPathOutcome.Unchanged ? JsonPathOutcome.Read : written.Outcome; + return (outcome, $"{readRules} + {written.Rule}", $"{readActions}; {written.Action}"); + } + private static string? KnownPolicyName(JsonObserverValueDelegate policy) { var declaring = policy.Method.DeclaringType; diff --git a/DragoAnt.System.Text.Json.Observer/Strategies/NameMatcher.cs b/DragoAnt.System.Text.Json.Observer/Strategies/NameMatcher.cs index cd7f396..bc945c7 100644 --- a/DragoAnt.System.Text.Json.Observer/Strategies/NameMatcher.cs +++ b/DragoAnt.System.Text.Json.Observer/Strategies/NameMatcher.cs @@ -1,4 +1,5 @@ using System.Text; +using System.Text.RegularExpressions; namespace DragoAnt.System.Text.Json.Observer.Strategies; @@ -30,11 +31,31 @@ private protected static StringComparison ComparisonOf(ref PropertyPath path) => public static NameMatcher OneOf(string[] names) => new OneOfNameMatcher(names); - internal sealed class FuncNameMatcher(Func match, string? description = null) : NameMatcher + internal sealed class FuncNameMatcher(Func match, string? description = null) : NameMatcher { + public FuncNameMatcher(Func match, string? description = null) + : this((name, _) => match(name), description) + { + } + public override string Describe() => description ?? "custom name test"; - public override bool MatchString(string? name, StringComparison comparison) => match(name); + public override bool MatchString(string? name, StringComparison comparison) => match(name, comparison); + } + + /// + /// A regular expression; a case-insensitive call also matches names that differ in case only. + /// + internal sealed class RegexNameMatcher(Regex regex) : NameMatcher + { + private readonly Regex _ignoreCase = (regex.Options & RegexOptions.IgnoreCase) != 0 + ? regex + : new Regex(regex.ToString(), regex.Options | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, regex.MatchTimeout); + + public override string Describe() => $"Regex(/{regex}/)"; + + public override bool MatchString(string? name, StringComparison comparison) => + name is not null && (comparison == StringComparison.Ordinal ? regex : _ignoreCase).IsMatch(name); } /// diff --git a/DragoAnt.System.Text.Json.Observer/Strategies/PropMatches.cs b/DragoAnt.System.Text.Json.Observer/Strategies/PropMatches.cs index 862f6c5..ea5b4fb 100644 --- a/DragoAnt.System.Text.Json.Observer/Strategies/PropMatches.cs +++ b/DragoAnt.System.Text.Json.Observer/Strategies/PropMatches.cs @@ -26,10 +26,12 @@ public static class PropMatches public static PropMatchingStrategy Contains(string value) => new(NameMatcher.Contains(value)); /// - /// Matches property name by regular expression. + /// Matches property name by regular expression. Like the other tests it follows + /// : by default a name that differs in case only also matches. + /// A regular expression built with ignores case under either option. /// /// Property name regular expression. - public static PropMatchingStrategy Regex(Regex regex) => new(new NameMatcher.FuncNameMatcher(v => v is not null && regex.IsMatch(v), $"Regex(/{regex}/)")); + public static PropMatchingStrategy Regex(Regex regex) => new(new NameMatcher.RegexNameMatcher(regex)); /// /// Matches property by full name equality. diff --git a/DragoAnt.System.Text.Json.Observer/Strategies/PropMatchingStrategy.cs b/DragoAnt.System.Text.Json.Observer/Strategies/PropMatchingStrategy.cs index 2deb61b..c6d7812 100644 --- a/DragoAnt.System.Text.Json.Observer/Strategies/PropMatchingStrategy.cs +++ b/DragoAnt.System.Text.Json.Observer/Strategies/PropMatchingStrategy.cs @@ -8,7 +8,8 @@ public readonly struct PropMatchingStrategy private readonly NameMatcher? _matcher; /// - /// Matches property names with a custom test; the name is decoded to a for it. + /// Matches property names with a custom test; the name is decoded to a for it. The test decides + /// case on its own; the overload that takes a follows . /// /// Name test; receives null for an array item. public PropMatchingStrategy(Func strategy) @@ -16,6 +17,19 @@ public PropMatchingStrategy(Func strategy) _matcher = new NameMatcher.FuncNameMatcher(strategy); } + /// + /// Matches property names with a custom test that honours ; + /// the name is decoded to a for it. + /// + /// + /// Name test; receives null for an array item, and or + /// as the call's case option. + /// + public PropMatchingStrategy(Func strategy) + { + _matcher = new NameMatcher.FuncNameMatcher(strategy); + } + internal PropMatchingStrategy(NameMatcher matcher) { _matcher = matcher; diff --git a/DragoAnt.System.Text.Json.Observer/TagMasking.cs b/DragoAnt.System.Text.Json.Observer/TagMasking.cs index b71f918..e8fd816 100644 --- a/DragoAnt.System.Text.Json.Observer/TagMasking.cs +++ b/DragoAnt.System.Text.Json.Observer/TagMasking.cs @@ -11,6 +11,19 @@ internal static class TagMasking /// Writes the strategy's replacement for the current value and moves past it; a container is never read. /// public static void Mask(ref Utf8JsonReader reader, JsonWriter writer, MaskTag tag, ref PropertyPath propPath) + { + writer.MaskOutput = true; + try + { + MaskValue(ref reader, writer, tag, ref propPath); + } + finally + { + writer.MaskOutput = false; + } + } + + private static void MaskValue(ref Utf8JsonReader reader, JsonWriter writer, MaskTag tag, ref PropertyPath propPath) { var options = writer.Options; var strategy = options.MaskStrategy ?? Utf8MaskStrategy.Default; diff --git a/README.md b/README.md index 5b1b4de..17910be 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ Build an observer once (a `static readonly` field) and share it: it is thread-sa ### Mask or extract -**Mask** rules (`MaskAny`, `MaskStr`, …) replace a value. **Read** rules (`ReadStr`, `ReadInt`, …) hand a value to a context object and write it unchanged. One observer can do both in the same pass: +**Mask** rules (`MaskAny`, `MaskStr`, …) replace a value. **Read** rules (`ReadStr`, `ReadInt`, …) hand a value to a context object; they do not decide what is written, so the default policy writes the value unless `.Unmasked()` or a mask method is chained on the read (`Match("ssn").ReadStr(f).MaskAny(MaskTag.Last4)`). One observer can do both in the same pass: ```csharp using DragoAnt.System.Text.Json.Observer; @@ -94,7 +94,7 @@ Console.WriteLine(JsonObserver.Obj(NullList).Mask(json)); ### Absolute and relative rules -**Absolute** rules follow the path from the root, one `Match` per level or several names in one `Match`. **Relative** rules (inside `Relative(...)`) match the end of a property's path at any depth. Names match case-insensitively; `PropMatches.StartsWith`, `EndsWith`, `Contains`, `OneOf` and `Regex` test a name differently. The first rule that matches wins. +**Absolute** rules follow the path from the root, one `Match` per level or several names in one `Match`. **Relative** rules (inside `Relative(...)`) match the end of a property's path at any depth. Names match case-insensitively; `PropMatches.StartsWith`, `EndsWith`, `Contains`, `OneOf` and `Regex` test a name differently and follow the same case option. The first rule that writes a value wins; read rules run on every match. ```csharp using DragoAnt.System.Text.Json.Observer; @@ -131,9 +131,9 @@ Every `Mask*` rule masks the **whole value whatever its JSON type** — a sensit | `MaskBool` | `true` / `false`, otherwise `null` | the function receives `null` | | `MaskAny(MaskTag)` | — the call's `Utf8MaskStrategy` writes `Full` `"***"`, `Last4` `"***1234"` (shorter than 8 characters: `"***"`), `Hash` `"hash:…"` or `Omit` `null` | stays `null` | | `Unmasked()` | — writes a string, number, boolean or `null` unchanged | | -| `ReadStr` / `ReadInt` / `ReadLong` / `ReadDecimal` / `ReadBool` / `ReadRaw` | hands the value to the context and writes it unchanged; a number that does not fit arrives as `null` | | +| `ReadStr` / `ReadInt` / `ReadLong` / `ReadDecimal` / `ReadBool` / `ReadRaw` | hands the value to the context; the default policy writes it unless `.Unmasked()` or a mask method follows on the same match; a number that does not fit arrives as `null` | | -The table holds for absolute and relative rules alike. A strategy is a constant string, a `Regex` whose matches become `*`, or a function of the value and the context; a `null` result writes `null`. A value longer than `MaxValueBytes` reaches the function cut to that length. `Hash` uses `JsonObserverOptions.HashKey`, or a random key per process when it is empty. +The table holds for absolute and relative rules alike. A strategy is a constant string, a `Regex` whose matches become `*`, or a function of the value and the context; a `null` result writes `null`. The function receives the whole value, and what it returns is never cut by `MaxValueBytes`, which limits values written unmasked only. `Hash` uses `JsonObserverOptions.HashKey`, or a random key per process when it is empty. ### Custom mask strategies @@ -274,13 +274,13 @@ Console.WriteLine($"{result.Status} {Encoding.UTF8.GetString(output.WrittenSpan) | Option | Default | Effect | | --- | --- | --- | | `MaxOutputBytes` | unlimited | output limit; when reached the output is closed and the status is `Truncated` | -| `MaxValueBytes` | unlimited | longest string written; a longer one is cut, ends with `…`, and the status is `Truncated` | +| `MaxValueBytes` | unlimited | longest string written unmasked; a longer one is cut, ends with `…`, and the status is `Truncated`; mask output is never cut | | `MaxDepth` | 64 | deeper nesting is `Invalid` | | `RelaxedEscaping` | `true` | non-ASCII and HTML characters are written unescaped | | `HashKey`, `MaskStrategy` | random per process, built-in | used by `MaskTag` rules | | `IgnoreNulls` | `false` | drops `null` properties and items, and objects and arrays left empty by that | | `Indented` | `false` | indented output | -| `PropertyNameCaseInsensitive` | `true` | match rule names and shapes ignoring case; pass the serializer's setting to match names as deserialization does | +| `PropertyNameCaseInsensitive` | `true` | match rule names, `PropMatches` tests (`Regex` included) and shapes ignoring case; pass the serializer's setting to match names as deserialization does | Input may contain comments and trailing commas; a UTF-8 byte order mark is skipped. Comments are not written. diff --git a/skills/json-observer-masking/SKILL.md b/skills/json-observer-masking/SKILL.md index 9a3c1b1..7f8b870 100644 --- a/skills/json-observer-masking/SKILL.md +++ b/skills/json-observer-masking/SKILL.md @@ -40,7 +40,7 @@ Console.WriteLine(masker.Mask("""{"user":"alice","password":"s3cret","card":{"nu 1. **Build once, share everywhere.** An observer is immutable and thread-safe; keep it in a `static readonly` field. Building one per call costs far more than masking. 2. **The default policy is `AllowList`.** A factory without a policy, and `Relative(rules)` without its second argument, write every string, number **and boolean** no rule names as `"***"` (`null` stays). Pass `BlockList` to keep unnamed values. 3. **Absolute vs relative.** Rules on a builder (`Obj(root => root.Match("order").Obj(…))`) follow the path from the root. Rules inside `Relative(…)` match the **end** of a path at any depth: `Match("card", "number")` hits every `…card.number`. An **array item is one path level**, so reach `{"lines":[{"qty":…}]}` with `Match("lines", anyItem, "qty")` where `anyItem = new PropMatchingStrategy(_ => true)`. The first rule that matches wins. -4. **Names match exactly and case-insensitively.** Use `PropMatches.EndsWith/StartsWith/Contains/OneOf/Regex` for anything else. +4. **Names match exactly and case-insensitively.** Use `PropMatches.EndsWith/StartsWith/Contains/OneOf/Regex` for anything else; they follow the same case option. 5. **Prefer `MaskAny` for secrets.** Every `Mask*` rule masks the whole value whatever its JSON type (a number, a boolean, an object), but `MaskAny` keeps `null` as `null` without calling your function, while `MaskStr` passes `null` to it. 6. **Tags for standard masks:** `MaskAny(MaskTag.Full)` → `"***"`, `Last4` → `"***1111"` (shorter than 8 characters → `"***"`), `Hash` → `"hash:<16 hex>"`, `Omit` → `null`. Set `JsonObserverOptions.HashKey` for hashes that correlate across processes; the default key is random per process. 7. **Never throws, always safe.** Both APIs return the masked prefix of cut-off or invalid input, closed into valid JSON, and never write a masked value in clear. `MaskStatus` is `Masked`, `Truncated`, `Invalid` or `NotJson` (empty output: empty input, or a root that is not an object or array). diff --git a/skills/json-observer-masking/examples.md b/skills/json-observer-masking/examples.md index e37fc59..e445e32 100644 --- a/skills/json-observer-masking/examples.md +++ b/skills/json-observer-masking/examples.md @@ -264,7 +264,7 @@ Console.WriteLine(JsonObserver.FromShape(shape).Mask(""" ## Extract values while masking -`JsonObserver.Obj(...)` adds `Read*` rules that hand a value to a context object and write it unchanged. The context-aware policies live in `JsonObserverValuePolicies`. `Mask(json, context)` masks and extracts in one pass; `Read(json, context)` only extracts and returns a `MaskResult`. +`JsonObserver.Obj(...)` adds `Read*` rules that hand a value to a context object. A read rule does not decide what is written: the default policy writes the value (here `BlockList`, so unchanged) unless `.Unmasked()` or a mask method is chained on the read, as in `Match("ssn").ReadStr(f).MaskAny(MaskTag.Last4)`. The context-aware policies live in `JsonObserverValuePolicies`. `Mask(json, context)` masks and extracts in one pass; `Read(json, context)` only extracts and returns a `MaskResult`. ```csharp using DragoAnt.System.Text.Json.Observer; @@ -297,4 +297,4 @@ sealed class OrderInfo } ``` -A number that does not fit the read type (a fraction for `ReadInt`, a 30-digit integer) reaches the callback as `null`; the token is still written unchanged. +A number that does not fit the read type (a fraction for `ReadInt`, a 30-digit integer) reaches the callback as `null`; the default policy still writes the token. diff --git a/skills/json-observer-masking/migrating-from-1x.md b/skills/json-observer-masking/migrating-from-1x.md index 4e83c20..250ec42 100644 --- a/skills/json-observer-masking/migrating-from-1x.md +++ b/skills/json-observer-masking/migrating-from-1x.md @@ -79,11 +79,11 @@ Console.WriteLine(observer.Mask("""{"cvv":123,"address":{"street":"Main 1"}}""") ## 5. A string cut by `MaxValueBytes` reports `Truncated` -It used to report success. `FailedAtByte` is -1 in that case (the whole document was read). Masking functions now receive such a value cut to `MaxValueBytes`. +It used to report success. `FailedAtByte` is -1 in that case (the whole document was read). The cap applies to values written unmasked only: see 12. ## 6. Number read rules no longer fail the body -`ReadInt`, `ReadLong` and `ReadDecimal` receive `null` for a number that does not fit, and the token is written unchanged. +`ReadInt`, `ReadLong` and `ReadDecimal` receive `null` for a number that does not fit; the default policy writes the token. ## 7. `PropertyPath` is a `ref struct` @@ -99,6 +99,35 @@ It cannot be derived from outside; `JsonWriter.FromUtf8JsonWriter`, `JsonWriter. ## 10. A UTF-8 byte order mark is skipped +## 11. Read rules no longer decide what is written + +A `Read*` rule used to write its value unchanged, even under `AllowList`. It now only hands the value to the context; the next rule on the same match or the default policy writes it. Where you relied on clear text, chain `.Unmasked()`; to read and mask one value, chain a mask method on the read: + +```csharp +using DragoAnt.System.Text.Json.Observer; +using DragoAnt.System.Text.Json.Observer.Strategies; + +var observer = JsonObserver.Obj(root => root + .Match("id").ReadInt((id, p) => p.Id = id).Unmasked() + .Match("ssn").ReadStr((ssn, p) => p.Ssn = ssn).MaskAny(MaskTag.Last4)); +var person = new Person(); +Console.WriteLine(observer.Mask("""{"id":7,"ssn":"123-45-6789","name":"Kim"}""", person)); +Console.WriteLine($"{person.Id} {person.Ssn}"); +// Output: +// {"id":7,"ssn":"***6789","name":"***"} +// 7 123-45-6789 + +sealed class Person +{ + public int? Id { get; set; } + public string? Ssn { get; set; } +} +``` + +## 12. Masking functions receive the whole value + +A masking function used to receive a value longer than `MaxValueBytes` cut to that length, so a `Last4`-style function printed digits from the middle; a hash was cut too. The function now receives the whole value, and mask output is never cut: `MaxValueBytes` limits values written unmasked only. + ## New in 2.0, worth adopting while you migrate - The UTF-8 API with a reused `IBufferWriter` ([recipes.md](./recipes.md#hot-path-utf-8-api)). diff --git a/skills/json-observer-masking/pitfalls.md b/skills/json-observer-masking/pitfalls.md index 5ac10ef..587580d 100644 --- a/skills/json-observer-masking/pitfalls.md +++ b/skills/json-observer-masking/pitfalls.md @@ -28,7 +28,7 @@ Check, in order: 1. **The rule is absolute, the field is nested.** `JsonObserver.Obj(root => root.Match("password")…)` only matches a top-level `password`. Use `Relative(...)` to match at any depth. 2. **The name differs.** Matching is exact (case-insensitive): `password` does not match `newPassword` or `passwd`. Use `PropMatches.Contains("password")` or `PropMatches.OneOf(...)`. 3. **The path crosses an array.** An array item is a path level: `Match("users", "password")` does not reach `{"users":[{"password":…}]}`. Use a single name in `Relative(...)`, or `Match("users", AnyItem, "password")` with `AnyItem = new PropMatchingStrategy(_ => true)`. -4. **The value is not a string and the rule only reads.** `Read*` rules write the value unchanged; use a `Mask*` rule. +4. **The rule only reads.** A `Read*` rule leaves the writing to the default policy, which under `BlockList` writes the value unchanged; chain a mask method on the read (`ReadStr(f).MaskAny(MaskTag.Full)`) or use a `Mask*` rule. ```csharp using DragoAnt.System.Text.Json.Observer; @@ -104,9 +104,9 @@ Console.WriteLine($"[{any.Mask("[1]", out var r3)}] {r3.Status}"); **Cause:** values shorter than 8 characters are masked fully, so a short value is not narrowed down. Booleans, objects and arrays are always `"***"`. -## A masking function sees a shortened value +## A masking function is slow on huge values -**Cause:** `MaxValueBytes` also cuts the value handed to a function. The result reports `Truncated`. +**Cause:** a masking function receives the whole value, decoded to a `string`, whatever `MaxValueBytes` says; the cap limits values written unmasked only. Prefer `MaskAny(MaskTag…)` or a constant for fields that can be huge: they never decode the value to a `string`. ## Building an observer per call diff --git a/skills/json-observer-masking/recipes.md b/skills/json-observer-masking/recipes.md index 3fc75a1..16b9692 100644 --- a/skills/json-observer-masking/recipes.md +++ b/skills/json-observer-masking/recipes.md @@ -117,7 +117,7 @@ A value that is cut never leaks: a string being written when the input ends is d | Option | Default | Effect | | --- | --- | --- | | `MaxOutputBytes` | unlimited | output cap in UTF-8 bytes; reaching it closes the output → `Truncated` | -| `MaxValueBytes` | unlimited | longest string written; a longer one is cut and ends with `…` → `Truncated`; masking functions also receive the cut value | +| `MaxValueBytes` | unlimited | longest string written unmasked; a longer one is cut and ends with `…` → `Truncated`; masking functions receive the whole value and their output is never cut | | `MaxDepth` | 64 | deeper nesting → `Invalid` | | `RelaxedEscaping` | `true` | non-ASCII and HTML characters written unescaped | | `HashKey` | random per process | key of `MaskTag.Hash` |