Remove every artificial throttle, repair the test oracles and the benchmark cells

The million-record CLUSTER ingestion gate never passed on a hosted runner:
ten SDK writers sharing one API key hit the per-principal HTTP admission
pool (eight concurrent requests per key per node) and got ResourceExhausted
or UnknownWriteOutcome within fifty seconds. The KeyLoad benchmark cells
stopped at exactly 100,000 records because the change outbox refused
producers once 100,000 entries were retained. The comparison harness also
gave KeyLoad nodes an admission profile that no other database had.
Measurements must show the real time of the work, so nothing refuses or
queues callers because other work is running.

Removed from request handling:
- HTTP admission (node/tenant/principal request counts, reserved control
  requests and bytes, heavy-read and working-set reservations, per-route
  body caps) and its KeyLoad:HttpAdmission section.
- Command admission (node/tenant/principal command counts, retained bytes,
  the runtime-journal bootstrap control lane) and KeyLoad:CommandAdmission.
- The MCP retained-memory pools and KeyLoad:McpMemory; the MCP request state
  keeps only the payload and reply ceilings.
- The admission status surface: GET /admin/admission, the SDK call, the
  keyload_admin_admission MCP tool, the read kind (NodeStatus keeps wire
  value 19), the grain/storage read methods and the dashboard card.
- Grain routing caps: request producers per silo, connections per node,
  operations per connection and the per-silo active work count.
  MaximumTotalFrames is now MaximumStreamFrames and only bounds the frames
  one long-running response stream carries.
- The Postgres cancellation registration count cap.
- The benchmark KeyLoad admission profile and its observations.

Removed from the engine:
- DatabaseLimits.MaxConcurrentQueries: analytical reads keep one counted
  reservation for release proofs, but the count never refuses.
- The change-outbox producer quota and projection progress reserve
  (MaxOutboxRecords/Bytes, ReservedOutboxRecords/Bytes and the consumer's
  LastProgressReservationCut); WriterQueueCapacity, which nothing used.
- The ANN reader caps per generation and slot and the ANN operation gate.
- TextExecutionOptions.MaximumActiveLeases: full-text readers, resource
  leases and maintenance sessions are counted for joined shutdown only.
  The bootstrap text projection keeps one reader per generation slot as a
  structural rule of its file lifecycle.
- The unused CommandInboxExecutionOptions type.

A duplicate active request or request-work identity now fails with
Conflict instead of ResourceExhausted. The request body keeps one
server-wide Kestrel limit (KeyLoad:ServerExecution:MaximumBodyBytes,
32 MiB); per-operation data bounds, lifetimes and memory/disk planning are
unchanged.

The heavy-load sentinel oracle now expects the read reference bound to the
collection's resource lifetime, as the server returns it.

Tests that only exercised removed limits are deleted with their helpers;
tests that used them as scaffolding now check the behavior that remains:
uncapped request work with Conflict on duplicates, hundreds of concurrent
analytical reservations, a concurrent reservation during a held text
search, the outbox head corruption flow without a quota write, projection
receipts across compaction and reopen, and competing text maintenance
sessions. The unit inventory, product contributors, docs (ADR-145 replaces
ADR-042) and status.json describe the current design.

Product defect found by the test sweep: PartitionRef cached its atomic
partition hash in a private field, and a `with` copy carried the original's
hash, so `partition with { PartitionKey = ... }` reported the wrong
AtomicPartitionId. The cache now records the identity components it was
computed from and recomputes when they differ; a regression test covers it.

Unit oracles brought up to the current server contracts:
- Store-image oracles exclude or expect the system last-command row that
  every command with a persisted outcome writes next to the applied
  watermark.
- Queue, event and header oracles expect the caller's exact JSON text,
  which the server preserves, instead of key-sorted copies; derived link
  payloads are expected in the server's QueueGraphLink field order.
- Duplicate request-work identities expect Conflict; the MCP catalog no
  longer counts the removed admission tool; ANN readers may overlap.
- Hosted Orleans fixtures get a 60-second start bound: every host start
  builds the MCP tool-search graph, whose full-text indexing is serialized
  process-wide in ManagedCode.MarkdownLd.Kb.

Packages: the owner's NuGet bump (ModelContextProtocol 2.3.0,
ManagedCode.Communication 10.3.15, Aspire 13.6.4, TUnit 1.76.0 and others)
is completed with ManagedCode.MCPGateway 2.3.0, released today on
ModelContextProtocol 2.3.0. MCP gateway invoke arguments are JsonElement
values; the new Communication analyzers' findings (redundant null guards,
negated IsSuccess) are fixed, including one latent benchmark check that
compared the read wrapper instead of its document with null.

Benchmarks:
- Target HTTP clients get the same 1 MiB response-header bound as document
  clients; the OpenSearch update-by-query backfill runs on the target
  client and returns about 230 KB of task headers.
- The MongoDB client runs on the host, where replica-set member names
  (container aliases) do not resolve, so it connects directly to the
  endpoints Aspire publishes for the primary and for each secondary.
- Neo4j cleanup deletes a million-node namespace in 10,000-node
  transactions within a 10-minute bound instead of one transaction.
- KeyLoad cleanup reads every identifier in one cursor pass and then
  deletes in batches; restarting the scan after each batch re-read every
  earlier tombstone.
- Build and Tests uploads TRX results for every suite and bounds the Aspire
  suite at 100 minutes so a slow run reports instead of being cancelled.

Verification: solution build with -warnaserror, dotnet format verify and the
governance check pass. The full unit project ran locally with ten parallel
tests before the oracle fixes (3641 cases, 135 failed, all known oracle,
analyzer-contract or load cases); after the fixes the previously failing
68 classes leave 10 failures at ten parallel tests, and all 10 pass when run
sequentially (MCP catalog build queueing and timing-sensitive ANN budget
cases under load). Integration, recovery and benchmark suites are left to
CI; the heavy-load gate was not rerun locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Author: KSemenenko

.github/workflows/build-and-tests.yml

@@ -27,13 +27,21 @@ jobs:
         shell: pwsh
         run: tests/KeyLoad.IntegrationTests/bin/Debug/net10.0/playwright.ps1 install --with-deps chromium
       - name: Test code analyzers
-        run: dotnet test --project tests/KeyLoad.Analyzers.Tests/KeyLoad.Analyzers.Tests.csproj --no-build --output Detailed
+        run: dotnet test --project tests/KeyLoad.Analyzers.Tests/KeyLoad.Analyzers.Tests.csproj --no-build --output Detailed --report-trx --results-directory "$GITHUB_WORKSPACE/TestResults/analyzers"
       - name: Test units
         if: ${{ !cancelled() }}
-        run: dotnet test --project tests/KeyLoad.UnitTests/KeyLoad.UnitTests.csproj --no-build --output Detailed --maximum-parallel-tests "$(nproc)"
+        run: dotnet test --project tests/KeyLoad.UnitTests/KeyLoad.UnitTests.csproj --no-build --output Detailed --maximum-parallel-tests "$(nproc)" --report-trx --results-directory "$GITHUB_WORKSPACE/TestResults/units"
       - name: Test process recovery
         if: ${{ !cancelled() }}
-        run: dotnet test --project tests/KeyLoad.RecoveryTests/KeyLoad.RecoveryTests.csproj --no-build --output Detailed
+        run: dotnet test --project tests/KeyLoad.RecoveryTests/KeyLoad.RecoveryTests.csproj --no-build --output Detailed --report-trx --results-directory "$GITHUB_WORKSPACE/TestResults/recovery"
       - name: Test Aspire topology
         if: ${{ !cancelled() }}
-        run: dotnet test --project tests/KeyLoad.IntegrationTests/KeyLoad.IntegrationTests.csproj --no-build --output Detailed --treenode-filter '/*/*/*/*[Category!=HeavyLoad]'
+        run: dotnet test --project tests/KeyLoad.IntegrationTests/KeyLoad.IntegrationTests.csproj --no-build --output Detailed --treenode-filter '/*/*/*/*[Category!=HeavyLoad]' --timeout 100m --report-trx --results-directory "$GITHUB_WORKSPACE/TestResults/integration"
+      - name: Save test results
+        if: ${{ always() }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: build-and-tests-results
+          path: TestResults/**
+          if-no-files-found: warn
+          retention-days: 30

Directory.Packages.props

@@ -7,18 +7,18 @@
   <ItemGroup>
     <PackageVersion Include="Cartograph" Version="0.1.0-alpha" />
     <PackageVersion Include="Cartograph.Catalog" Version="0.1.0-alpha" />
-    <PackageVersion Include="ZoneTree" Version="2.0.2" />
+    <PackageVersion Include="ZoneTree" Version="2.0.3" />
     <PackageVersion Include="ZoneTree.Analytics" Version="1.0.2" />
     <PackageVersion Include="ZoneTree.FullTextSearch" Version="1.1.0" />
-    <PackageVersion Include="ManagedCode.Communication" Version="10.3.11" />
-    <PackageVersion Include="ManagedCode.Communication.AspNetCore" Version="10.3.11" />
-    <PackageVersion Include="ManagedCode.Communication.Orleans" Version="10.3.11" />
+    <PackageVersion Include="ManagedCode.Communication" Version="10.3.15" />
+    <PackageVersion Include="ManagedCode.Communication.AspNetCore" Version="10.3.15" />
+    <PackageVersion Include="ManagedCode.Communication.Orleans" Version="10.3.15" />
     <PackageVersion Include="ManagedCode.TimeSeries" Version="10.1.2" />
     <PackageVersion Include="ManagedCode.Storage.FileSystem" Version="10.1.5" />
     <PackageVersion Include="ManagedCode.Storage.Server" Version="10.1.5" />
     <PackageVersion Include="ManagedCode.Orleans.Graph" Version="10.4.7" />
     <PackageVersion Include="ManagedCode.Orleans.Identity.Core" Version="10.4.2" />
-    <PackageVersion Include="ManagedCode.MCPGateway" Version="0.4.24" />
+    <PackageVersion Include="ManagedCode.MCPGateway" Version="2.3.0" />
     <PackageVersion Include="ManagedCode.MarkdownLd.Kb" Version="0.2.17" />
     <!-- Central transitive dependency used by Aspire and StreamJsonRpc. -->
     <PackageVersion Include="MessagePack" Version="3.1.12" />
@@ -29,8 +29,8 @@
     <PackageVersion Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.13" />
     <PackageVersion Include="Microsoft.SqlServer.TransactSql.ScriptDom" Version="180.140.0" />
     <PackageVersion Include="System.IO.Hashing" Version="10.0.13" />
-    <PackageVersion Include="ModelContextProtocol.AspNetCore" Version="2.2.0" />
-    <PackageVersion Include="ModelContextProtocol.Core" Version="2.2.0" />
+    <PackageVersion Include="ModelContextProtocol.AspNetCore" Version="2.3.0" />
+    <PackageVersion Include="ModelContextProtocol.Core" Version="2.3.0" />
     <PackageVersion Include="Microsoft.Orleans.Server" Version="10.4.0" />
     <PackageVersion Include="ManagedCode.Orleans.RapidMembership" Version="10.4.0" />
     <PackageVersion Include="Microsoft.Orleans.Streaming" Version="10.4.0" />
@@ -45,30 +45,30 @@
     <PackageVersion Include="Microsoft.CodeAnalysis.Common" Version="5.9.0" />
     <PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="5.9.0" />
     <PackageVersion Include="Microsoft.CodeAnalysis.Workspaces.Common" Version="5.9.0" />
-    <PackageVersion Include="Aspire.Hosting.AppHost" Version="13.6.2" />
-    <PackageVersion Include="Aspire.Hosting.Testing" Version="13.6.2" />
-    <PackageVersion Include="Aspire.Hosting.PostgreSQL" Version="13.6.2" />
-    <PackageVersion Include="Aspire.Hosting.Qdrant" Version="13.6.2" />
-    <PackageVersion Include="Aspire.Hosting.RabbitMQ" Version="13.6.2" />
-    <PackageVersion Include="Aspire.Hosting.Redis" Version="13.6.2" />
+    <PackageVersion Include="Aspire.Hosting.AppHost" Version="13.6.4" />
+    <PackageVersion Include="Aspire.Hosting.Testing" Version="13.6.4" />
+    <PackageVersion Include="Aspire.Hosting.PostgreSQL" Version="13.6.4" />
+    <PackageVersion Include="Aspire.Hosting.Qdrant" Version="13.6.4" />
+    <PackageVersion Include="Aspire.Hosting.RabbitMQ" Version="13.6.4" />
+    <PackageVersion Include="Aspire.Hosting.Redis" Version="13.6.4" />
     <PackageVersion Include="Microsoft.Extensions.Http.Resilience" Version="10.11.0" />
     <PackageVersion Include="Microsoft.Extensions.ServiceDiscovery" Version="10.11.0" />
     <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.20.1" />
     <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.20.1" />
-    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.20.0" />
-    <PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.20.0" />
-    <PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="1.20.0" />
-    <PackageVersion Include="TUnit" Version="1.75.0" />
-    <PackageVersion Include="TUnit.Aspire" Version="1.75.0" />
-    <PackageVersion Include="Microsoft.Playwright" Version="1.63.0" />
+    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.20.1" />
+    <PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.20.1" />
+    <PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="1.20.1" />
+    <PackageVersion Include="TUnit" Version="1.76.0" />
+    <PackageVersion Include="TUnit.Aspire" Version="1.76.0" />
+    <PackageVersion Include="Microsoft.Playwright" Version="1.64.1" />
     <PackageVersion Include="Microsoft.FluentUI.AspNetCore.Components" Version="5.0.1" />
     <!-- One Microsoft engine: MTP collects test processes; the same-version CLI collects owned CLUSTER containers. -->
     <PackageVersion Include="Microsoft.Testing.Extensions.CodeCoverage" Version="18.12.0" />
     <PackageVersion Include="BenchmarkDotNet" Version="0.15.8" />
     <PackageVersion Include="Npgsql" Version="10.0.3" />
     <PackageVersion Include="RabbitMQ.Client" Version="7.2.2" />
-    <PackageVersion Include="StackExchange.Redis" Version="3.4.5" />
-    <PackageVersion Include="MongoDB.Driver" Version="3.12.0" />
+    <PackageVersion Include="StackExchange.Redis" Version="3.4.20" />
+    <PackageVersion Include="MongoDB.Driver" Version="3.13.0" />
     <PackageVersion Include="KurrentDB.Client" Version="1.6.0" />
   </ItemGroup>
 </Project>

benchmarks/KeyLoad.BenchmarkScenarios/Benchmarks/EmbeddedBenchmarks.cs

@@ -196,7 +196,7 @@ private void Submit<T>(OperationKind kind, T payload)
             var result = Database.Apply(operation);
             if (result.IsFailed)
             {
-                throw Errors.Fail(Errors.Code(result.Problem!)!.Value, result.Problem!.Detail!);
+                throw Errors.Fail(Errors.Code(result.Problem)!.Value, result.Problem.Detail!);
             }
         }
     }

benchmarks/KeyLoad.ComparisonHost/Configuration/ComparisonExecutionRegistration.cs

@@ -59,17 +59,6 @@ internal static IOptions<ComparisonHostExecutionOptions> ReadHost(IConfiguration
         });
     }
 
-    internal static IOptions<IsolatedKeyLoadAdmissionOptions> ReadIsolatedAdmission(IConfiguration configuration)
-    {
-        var options = new OptionsManager<IsolatedKeyLoadAdmissionOptions>(new OptionsFactory<IsolatedKeyLoadAdmissionOptions>(
-            [new ConfigureFromConfigurationOptions<IsolatedKeyLoadAdmissionOptions>(
-                configuration.GetSection(IsolatedKeyLoadAdmissionOptions.SectionName))], [],
-            [new ValidateOptions<IsolatedKeyLoadAdmissionOptions>(Options.DefaultName,
-                value => value.IsValid(), IsolatedKeyLoadAdmissionOptions.ValidationMessage)]));
-        _ = options.Value;
-        return options;
-    }
-
     internal static IOptions<KeyLoadClientExecutionOptions> ReadClient(IConfiguration configuration)
     {
         var options = new OptionsManager<KeyLoadClientExecutionOptions>(new OptionsFactory<KeyLoadClientExecutionOptions>(

benchmarks/KeyLoad.ComparisonHost/Hosting/IsolatedHostTargetOwner.cs

@@ -10,7 +10,6 @@ namespace KeyLoad.ComparisonHost.BenchmarkComparisons;
 /// <summary>Owns only one execution target and tracks partial HTTP construction until ownership transfers.</summary>
 internal sealed class IsolatedHostTargetOwner(IOptions<ComparisonExecutionOptions> executionOptions,
     IOptions<ComparisonDiagnosticOptions> diagnosticOptions,
-    IOptions<IsolatedKeyLoadAdmissionOptions> admissionOptions,
     IOptions<ComparisonLifecycleOptions> lifecycleOptions, IOptions<KeyLoadClientExecutionOptions> clientOptions, IOptions<QueryTranslationOptions> translationOptions, TimeProvider? provider = null) : IAsyncDisposable
 {
     private const string ExternalImageRequired = "An external database comparison requires its configured container image.";
@@ -20,6 +19,7 @@ internal sealed class IsolatedHostTargetOwner(IOptions<ComparisonExecutionOption
     private ComparisonExecutionOptions Policy => execution.Value;
     internal IOptions<ComparisonExecutionOptions> ExecutionOptions => execution;
     private readonly List<HttpClient> unownedClients = [];
+    private readonly List<SocketsHttpHandler> clientHandlers = [];
     private IComparisonTarget? target;
 
     internal TargetProfile? ObservedProfile => target?.Profile;
@@ -56,7 +56,7 @@ private void InitializeTarget(IsolatedHostSettings settings)
             IsolatedHostConstants.Rabbit => new RabbitTarget(execution.Connection!, settings.RunId, RequireExternalImage(execution), lifecycleOptions, topology, CreateClient(execution, PrimaryEndpointIndex), provider: Clock),
             IsolatedHostConstants.Redis => new RedisTarget(execution.Connection!, settings.RunId, RequireExternalImage(execution), lifecycleOptions, executionOptions, diagnostics, topology, [.. execution.Replicas], provider: Clock),
             IsolatedHostConstants.Neo4j => new Neo4jTarget(CreateClient(execution, PrimaryEndpointIndex), settings.RunId, RequireExternalImage(execution), lifecycleOptions, executionOptions, provider: Clock),
-            IsolatedHostConstants.Mongo => new MongoTarget(execution.Connection!, settings.RunId, RequireExternalImage(execution), topology, lifecycleOptions, executionOptions, provider: Clock),
+            IsolatedHostConstants.Mongo => new MongoTarget(execution.Connection!, execution.Endpoints, settings.RunId, RequireExternalImage(execution), topology, lifecycleOptions, executionOptions, provider: Clock),
             IsolatedHostConstants.OpenSearch => new OpenSearchTarget(CreateClient(execution, PrimaryEndpointIndex), settings.RunId, RequireExternalImage(execution), topology, lifecycleOptions, executionOptions, provider: Clock),
             IsolatedHostConstants.Kurrent => new KurrentTarget(execution.Connection!, CreateClients(execution), settings.RunId, RequireExternalImage(execution), topology, lifecycleOptions, diagnostics, provider: Clock),
             _ => throw new InvalidOperationException(IsolatedHostConstants.Failure)
@@ -73,10 +73,7 @@ private KeyLoadTarget CreateKeyLoad(IsolatedHostSettings settings, RuntimeIsolat
         const int ClientsFirstIndex = 0;
 
         var clients = CreateClients(execution);
-        return new(clients[ClientsFirstIndex], execution.AdminKey!, settings.RunId, lifecycleOptions, executionOptions, diagnostics, admissionOptions, clientOptions, translationOptions, execution.Image, clients, settings.Selection.NodeCount, provider: Clock)
-        {
-            RequireIsolatedAdmission = true
-        };
+        return new(clients[ClientsFirstIndex], execution.AdminKey!, settings.RunId, lifecycleOptions, executionOptions, diagnostics, clientOptions, translationOptions, execution.Image, clients, settings.Selection.NodeCount, provider: Clock);
     }
 
     private QdrantTarget CreateQdrant(IsolatedHostSettings settings, RuntimeIsolatedHostSettings execution)
@@ -101,7 +98,16 @@ private HttpClient[] CreateClients(RuntimeIsolatedHostSettings execution)
 
     private HttpClient CreateClient(RuntimeIsolatedHostSettings execution, int index)
     {
-        var client = new HttpClient { BaseAddress = execution.Endpoints[index], Timeout = Timeout.InfiniteTimeSpan };
+        var handler = new SocketsHttpHandler
+        {
+            MaxResponseHeadersLength = this.execution.Value.DocumentClientMaxResponseHeadersKilobytes
+        };
+        clientHandlers.Add(handler);
+        var client = new HttpClient(handler, disposeHandler: false)
+        {
+            BaseAddress = execution.Endpoints[index],
+            Timeout = Timeout.InfiniteTimeSpan
+        };
         unownedClients.Add(client);
         if (execution.User is not null)
         {
@@ -134,6 +140,11 @@ public async ValueTask DisposeAsync()
                 client.Dispose();
             }
             unownedClients.Clear();
+            foreach (var handler in clientHandlers)
+            {
+                handler.Dispose();
+            }
+            clientHandlers.Clear();
         }
     }
 

benchmarks/KeyLoad.ComparisonHost/Session/Fixtures/BenchmarkSessionFixture.cs

@@ -67,7 +67,6 @@ private void InitializeCore()
         {
             owner = new(ComparisonExecutionRegistration.Read(input),
                 ComparisonExecutionRegistration.ReadDiagnostics(input),
-                ComparisonExecutionRegistration.ReadIsolatedAdmission(input),
                 ComparisonExecutionRegistration.ReadLifecycle(input),
                 ComparisonExecutionRegistration.ReadClient(input),
                 ComparisonExecutionRegistration.ReadTranslation(input), TimeProvider.System);

benchmarks/KeyLoad.Comparisons/Configuration/ComparisonExecutionOptions.cs

@@ -10,7 +10,7 @@ public sealed partial class ComparisonExecutionOptions
     public int DocumentVerificationConnectionReserve { get; set; } = RequiredDocumentClientConnections;
     /// <summary>One actual connection per independently admitted HTTP document client.</summary>
     public int DocumentClientMaxConnections { get; set; } = RequiredDocumentClientConnections;
-    /// <summary>Response header bound of the document HTTP client; OpenSearch answers a synchronous update-by-query with one TASK_RESOURCE_USAGE header per scroll batch, about 230 KB for one million documents.</summary>
+    /// <summary>Response header bound of every target and document HTTP client; OpenSearch answers a synchronous update-by-query with one TASK_RESOURCE_USAGE header per scroll batch, about 230 KB for one million documents.</summary>
     public int DocumentClientMaxResponseHeadersKilobytes { get; set; } = RequiredDocumentClientResponseHeadersKilobytes;
     /// <summary>Joined cleanup bound for a complete million-record execution document namespace.</summary>
     public TimeSpan DocumentCleanupTimeout { get; set; } = RequiredDocumentCleanupTimeout;

benchmarks/KeyLoad.Comparisons/Configuration/ComparisonLifecycleOptions.cs

@@ -21,7 +21,7 @@ public sealed class ComparisonLifecycleOptions
     private const int DiagnosticSeconds = 2;
     private const int ReceiptSeconds = 3;
     private const int ProbeExpiryMinutes = 2;
-    private const int Neo4jCleanupSeconds = 15;
+    private const int Neo4jCleanupSeconds = 600;
     private const int MongoCleanupSeconds = 15;
     private const int QdrantCleanupSeconds = 10;
     private const int OpenSearchCleanupSeconds = 10;