Skip to content

Commit 2ff93ad

Browse files
committed
Join native fixture ownership and port current connection lifecycle checks
Commit the complete current working-tree scope. Give the existing shared cluster native memoized async disposal and explicit port ownership, borrow OrleansNode from DI, and reuse the same real two-silo hosts from connection, persisted outcome and generic application-disposal checks. Preserve original failures, native signing/stream calls, canonical versus provider cuts, file release and cold continuations. Validation: current product dependencies passed strict Release builds; RecoveryTests strict Release build passed with 0 warnings/errors; focused source syntax and diff checks passed. The existing real analytics process recovery case failed before its startup acknowledgement, with all child cleanup joined; startup diagnosis remains open. Full Unit/Integration/solution and distributed qualification remain pending.
1 parent 1773839 commit 2ff93ad

26 files changed

Lines changed: 291 additions & 271 deletions
Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,40 @@
11
# Generic application disposal after native partial startup
22

3-
This native regression covers [ADR-136](../../ADR/ADR-136-native-orleans-partition-ownership.md) and the ownership contract in [StorageRecovery](../StorageRecovery.md). Existing signed connection semantics remain [ADR-125](../../ADR/ADR-125-connection-owned-execution.md); native TUnit ownership follows [ADR-117](../../ADR/ADR-117-native-tunit-ci-entry.md).
3+
This native case covers [ADR-136](../../ADR/ADR-136-native-orleans-partition-ownership.md) and the ownership contract in [StorageRecovery](../StorageRecovery.md). Connection ownership follows [ADR-125](../../ADR/ADR-125-connection-owned-execution.md), and native TUnit ownership follows [ADR-117](../../ADR/ADR-117-native-tunit-ci-entry.md).
44

55
## Operation and independent oracles
66

7-
The case calls the existing production `ServerConfiguration.BuildAsync` with the original native localhost selection and physical-owner settings. It creates no alternate host builder, provider or transport. The same native ApplicationServices observation performs the real second `node.owner.lock` acquisition after successful native storage Init. The original startup task must fail with that exact actual IOException. Before/after complete bounded canonical records, full native identity and physical position are compared inside the same original store read gate. Unexpected startup errors are preserved before proof errors; they are not replaced by a missing-observation assertion.
7+
The case calls production `ServerConfiguration.BuildAsync` with the shared `RequestCqrsHostedProfile` and native localhost membership. One silo and one persisted copy are configured. The original arguments, native identities, physical directory and port are frozen for all applications. The native ApplicationServices participant performs a real second `node.owner.lock` acquisition after successful storage Init. The original startup task must fail with that exact IOException. Complete bounded canonical records, full storage identity and physical position are compared before and after the refusal under the same storage read gate. Unexpected startup errors remain before proof errors.
88

9-
Before invalidating the provider, the test captures the actual owner, borrowed node, registry and the three actual native grain services. Two authoritative files—`identity.json` and `commands.wal`—are read with an aggregate bound derived from the original native database profile. Actual FileStream reads allocate only their bounded declared lengths, reject an additional byte, retain the original token, and record read and separate disposal failures in order. Private bytes are compared as booleans, never emitted in assertions. Disposable native tree files are not an invariant.
9+
Before invalidating the provider, the test captures the actual physical owner, node and connection registry. The canonical store's `identity.json` and `commands.wal` are read with one aggregate byte bound from the actual database profile. FileStream reads allocate only their bounded declared lengths, reject an additional byte, retain the caller token, and preserve read and independent reader-disposal failures. Private bytes are compared as booleans and never emitted in assertions. Derived native tree files are not an invariant.
1010

11-
The only shutdown call initiating cleanup is the original `WebApplication.DisposeAsync`. There is no preceding application Stop, borrowed-node Stop, registry shutdown, producer join or fixture wrapper close. The terminal owner must perform registry → node → three native producers → physical host cleanup. The test requires its captured original Closing and node Closing to succeed, full actual `CanReleaseRoot` and joined node work, and exactly three retained original task references. Only after completed disposal does it compare these references with each actual native service's already-fenced JoinStorageProducerAsync result. It does not invoke registry methods to repair missing cleanup. Actual release is additionally tested by reacquiring the original OS node lock, with the returned handle disposed independently; both authoritative files must remain byte-identical.
11+
The only shutdown operation initiating cleanup is the original `WebApplication.DisposeAsync`. No application Stop, node Stop, registry shutdown or storage-lifecycle call precedes it. Actual owner Closing and node Closing must succeed, native request work and the complete node lifetime must be joined, and `CanReleaseRoot` must be true. The test does not invoke registry methods to repair missing cleanup. After successful disposal and release permission, it reacquires the existing OS node lock and disposes that handle independently. Both canonical authority files must remain byte-identical.
1212

13-
The same original root/settings then enter the existing native multi-node fixture. Unchanged reviewed helpers perform a genuine signed ConfigureResource and Batch write, signed minimum-token read, full independent process-durable receipt, empty private CompositionReferences, exact replay and two actual same-root restarts. Full identity and whole literal document/receipt oracles repeat after both restarts. Original body, file reader, direct disposal and assertion errors retain ordering. Unknown ownership or any failure retains the directory and attaches the existing bounded failure-root evidence; no cleanup success is inferred from a faulted task.
13+
The same original arguments and root then enter the shared native healthy-operation helper. Native ConfigureResource and Batch commands, a connection-grain CQRS minimum read and exact replay must preserve the independent complete `ProcessDurable` receipt, one persisted copy, empty private CompositionReferences, whole literal document and raw persisted StoredOutcome bytes. Replay cannot advance the canonical or physical cut. Two actual cold applications repeat full storage identity, receipt, outcome, minimum-read and release assertions. Original body, file-reader, direct-disposal and assertion errors remain observable. Any failure retains the directory and attaches its bounded root path to the original failure; a faulted cleanup task never implies release.
1414

1515
```mermaid
1616
sequenceDiagram
1717
participant App as Same production application
1818
participant GS as Native storage Init
1919
participant Trial as Test observation at ApplicationServices
2020
participant Owner as Actual terminal physical owner
21-
participant FreshCohort as Fresh same-root native cluster applications
21+
participant Fresh as Same-root native applications
2222
App->>GS: Native startup
2323
GS->>Owner: Original opening
2424
Trial->>Owner: Real held-lock acquisition refusal
2525
App-->>Trial: Original startup failure
2626
Trial->>App: ONLY original DisposeAsync
2727
App->>Owner: Generic DI terminal disposal
28-
Owner->>Owner: Registry, node, producers, files
28+
Owner->>Owner: Registry, native requests, storage lifetime
2929
Owner-->>Trial: Original joins and actual release permit
3030
Trial->>Trial: OS lock reacquisition and file conservation
31-
Trial->>FreshCohort: Signed healthy/replay/two cold
31+
Trial->>Fresh: Native healthy operation, replay and two cold applications
3232
```
3333

3434
## Qualification boundary
3535

36-
Mandatory complete case: `PartitionStorageNativeApplicationDisposeTests.ActualGenericApplicationDisposeAfterNativePartialStartupRetainsOwnersThenSignedReceiptThroughTwoColdStarts`.
36+
Existing mandatory case: `PartitionStorageNativeApplicationDisposeTests.ActualGenericApplicationDisposeAfterNativePartialStartupRetainsOwnersThenCanonicalReceiptThroughTwoColdStarts`.
3737

38-
The fault occurs at stage10000 before Active20000. Native GrainService.Start/background producers have not started; the three genuine retained join results may therefore be CompletedTask for never-started producers. This case does **not** qualify LIVE producer shutdown; the distinct `PartitionStorageNativeProducerDispose*` family owns that gate. Nor does this prove native `SiloTerminated`: pinned native Silo.Dispose suppresses the Starting-state Stop refusal, unlike explicit hosted application Stop. The separate wrapper partial-start case owns its original native Stop-error oracle.
38+
The fault occurs during ApplicationServices before Active. The case does not prove shutdown of active background stateless-worker work; the distinct `PartitionStorageNativeProducerDispose*` family owns that gate. Native silo termination is also separate from the physical-owner release permit. The explicit partial-start Stop case in [PartitionStorageNativeStartup](PartitionStorageNativeStartup.md) owns the exact native Starting-state Stop-error oracle. This case observes only the original generic disposal and its real terminal outcomes.
3939

40-
These are real in-process native server/physical-group operations, not Aspire Docker cluster or public SDK/official MCP qualification. Linux runtime, native OS behavior, deadline fit and the broader recovery/endurance gates are separate. Original tokens, limits, resource ownership and the existing native restart/shutdown budgets are unchanged.
40+
This source adaptation has not yet been compiled or run. It describes native single-silo production composition and physical-file checks. Aspire cluster operations, public SDK/official MCP interoperability, abrupt process recovery, power loss, Linux runtime, native OS behavior, deadline fit and broader recovery/endurance remain independent open gates. The configured startup timing, original cancellation, resource limits and joined cleanup remain required.

‎tests/KeyLoad.RecoveryTests/ClusterRouting/Fixtures/RequestExecution/Activation/RequestCqrsHostedRuntimeFixture.cs‎

Lines changed: 24 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,6 @@ internal sealed class RequestCqrsHostedRuntimeFixture : IAsyncInitializer, IAsyn
2424
private readonly TestClusterPortAllocator ports = new();
2525
private readonly string directory = Path.Combine(Path.GetTempPath(), DirectoryPrefix + Guid.NewGuid().ToString(GuidFormat));
2626
private WebApplication? application;
27-
private OrleansNode? node;
2827
private string[]? arguments;
2928
private int? reservedSiloPort;
3029
private Task? stopping;
@@ -36,10 +35,12 @@ internal RequestCqrsHostedRuntimeFixture(TimeProvider? clock = null, GrainRoutin
3635
bool nativeTextMaintenance = false, bool nativeAnnMaintenance = false, DatabaseLimits? limits = null,
3736
RequestCqrsHostedClusterNode? clusterNode = null,
3837
Func<IServiceProvider, ILifecycleParticipant<ISiloLifecycle>>? lifecycleParticipantFactory = null,
39-
IDatabaseAuthenticationExecutionObserver? authenticationExecutionObserver = null)
38+
IDatabaseAuthenticationExecutionObserver? authenticationExecutionObserver = null,
39+
Action<IServiceCollection, NodeOptions>? configureServices = null)
4040
{
4141
profile = new(clock ?? TimeProvider.System, RequestCqrsHostedProfile.ValidateRouting(routing), observer, purposeObserver,
42-
nativeTextMaintenance, nativeAnnMaintenance, limits, clusterNode, lifecycleParticipantFactory, authenticationExecutionObserver);
42+
nativeTextMaintenance, nativeAnnMaintenance, limits, clusterNode, lifecycleParticipantFactory,
43+
authenticationExecutionObserver, configureServices);
4344
}
4445

4546
internal IGrainFactory Grains => Services.GetRequiredService<IGrainFactory>();
@@ -50,7 +51,7 @@ internal RequestCqrsHostedRuntimeFixture(TimeProvider? clock = null, GrainRoutin
5051
internal IServiceProvider Services => application?.Services ?? throw new InvalidOperationException(NotInitialized);
5152
private PartitionHost Local => Services.GetRequiredService<PartitionHostApplicationOwner>().Borrow();
5253
internal GrainRequestCodec Codec => Services.GetRequiredService<GrainRequestCodec>();
53-
internal NativeRequestWorkOwner RequestWork => (node ?? throw new InvalidOperationException(NotInitialized)).RequestWork;
54+
internal NativeRequestWorkOwner RequestWork => Services.GetRequiredService<OrleansNode>().RequestWork;
5455
internal NativeTextIncrementalMaintenanceService? NativeMaintenance => profile.NativeText ? Local.TextMaintenance : null;
5556
internal NativeAnnMaintenanceService? NativeAnnMaintenance => profile.NativeAnn ? Local.AnnMaintenance : null;
5657
internal RequestCqrsHostedDatabase Database { get; private set; } = null!;
@@ -128,7 +129,7 @@ private async Task StartApplicationAsync(CancellationToken cancellationToken)
128129
lifecycleParticipantFactory: profile.LifecycleParticipantFactory,
129130
phaseObserver: profile.Observer, purposeObserver: profile.PurposeObserver,
130131
configureServices: profile.Register, startupCancellation: startup.Token).ConfigureAwait(false);
131-
node = Services.GetRequiredService<OrleansNode>();
132+
var node = Services.GetRequiredService<OrleansNode>();
132133
await application.StartAsync(startup.Token).ConfigureAwait(false);
133134
await node.StartAsync(startup.Token).ConfigureAwait(false);
134135
await Local.Coordinator.ReadBarrierAsync(startup.Token).ConfigureAwait(false);
@@ -149,7 +150,18 @@ private async Task DisposeCoreAsync()
149150
await ServerFailureObserver.ObserveAsync(() => StopAsync(CancellationToken.None), failures).ConfigureAwait(false);
150151
if (failures.Count == NoFailures)
151152
{
152-
ServerFailureObserver.Observe(ports.Dispose, failures);
153+
try
154+
{
155+
ports.Dispose();
156+
}
157+
catch (Exception error) when (NativeCqrsBoundaryErrors.IsNonFatal(error))
158+
{
159+
failures.Add(error);
160+
}
161+
catch (Exception error) when (!NativeCqrsBoundaryErrors.IsNonFatal(error))
162+
{
163+
failures.Add(error);
164+
}
153165
if (!Volatile.Read(ref retainDirectory) && Directory.Exists(directory))
154166
{
155167
ServerFailureObserver.Observe(() => Directory.Delete(directory, recursive: true), failures);
@@ -168,7 +180,6 @@ private async Task StopApplicationAsync()
168180
await CloseApplicationAsync(application, failures).ConfigureAwait(false);
169181
ServerFailureObserver.ThrowIfAny(failures);
170182
application = null;
171-
node = null;
172183
Database = null!;
173184
}
174185

@@ -178,15 +189,13 @@ private async Task CloseApplicationAsync(WebApplication app, List<Exception> fai
178189
var connections = Services.GetRequiredService<ServerConnectionRegistry>();
179190
await ServerFailureObserver.ObserveAsync(() => connections.ShutdownAsync(deadline.Token), failures).ConfigureAwait(false);
180191
await ServerFailureObserver.ObserveAsync(() => connections.ShutdownAsync(CancellationToken.None), failures).ConfigureAwait(false);
181-
if (node is not null)
192+
var node = app.Services.GetRequiredService<OrleansNode>();
193+
await ServerFailureObserver.ObserveAsync(() => node.StopAsync(deadline.Token), failures).ConfigureAwait(false);
194+
await ServerFailureObserver.ObserveAsync(() => node.StopAsync(CancellationToken.None), failures).ConfigureAwait(false);
195+
if (!node.HasJoinedRequestWork)
182196
{
183-
await ServerFailureObserver.ObserveAsync(() => node.StopAsync(deadline.Token), failures).ConfigureAwait(false);
184-
await ServerFailureObserver.ObserveAsync(() => node.StopAsync(CancellationToken.None), failures).ConfigureAwait(false);
185-
if (!node.HasJoinedRequestWork)
186-
{
187-
failures.Add(new InvalidOperationException(Unjoined));
188-
return;
189-
}
197+
failures.Add(new InvalidOperationException(Unjoined));
198+
return;
190199
}
191200
await ServerFailureObserver.ObserveAsync(() => app.StopAsync(deadline.Token), failures).ConfigureAwait(false);
192201
await ServerFailureObserver.ObserveAsync(() => Services.GetRequiredService<PartitionStorageLifecycle>()

0 commit comments

Comments
 (0)