Skip to content

Commit cfc0b4b

Browse files
committed
Fix due policy retries and use Aspire-owned HTTPS endpoints
Bind stable due command IDs to the fresh principal policy epoch. Observe completed autonomous transfers and validate changed query identities across cold restarts. Replace manual public ports and the cleartext HTTP2 listener with one Aspire-allocated HTTPS endpoint, including real HTTP2 response checks and CI certificate trust. Validation: restore, dotnet format, repository governance and shard partition checks passed. Local full build is blocked by concurrent uncommitted KLD0040 analyzer changes; complete Linux Build and Tests qualification is required.
1 parent e6606a2 commit cfc0b4b

31 files changed

Lines changed: 150 additions & 453 deletions

File tree

‎.github/workflows/build-and-tests.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,8 @@ jobs:
123123
uses: actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v6.0.0
124124
with:
125125
global-json-file: global.json
126+
- name: Trust the development HTTPS certificate
127+
run: dotnet dev-certs https --trust
126128
- name: Build
127129
run: dotnet build KeyLoad.slnx -warnaserror
128130
- name: Install administration browser

‎README.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -259,7 +259,8 @@ still serve earlier committed data after Orleans membership settles.
259259

260260
Both values are read at startup; there is no live resizing or data movement.
261261
Node data remains in `data/cluster/`; development credentials live in AppHost user secrets.
262-
HTTP ports start at `5101`; keep the data directory between restarts.
262+
Aspire allocates HTTPS, silo and membership ports; copy a node's HTTPS URL from
263+
the Aspire dashboard into `KEYLOAD_ENDPOINT`. Keep the data directory between restarts.
263264
Stop owned resources with `aspire stop --non-interactive`.
264265

265266
For one shared volume, configure the same storage path on all silos. The execution
@@ -276,11 +277,11 @@ shared filesystem locking and flush behavior still need deployment qualification
276277

277278
Export the `Parameters:admin-key` value from AppHost user secrets as `KEYLOAD_API_KEY` before running the CLI or the examples below.
278279

279-
- Open the [admin console](http://localhost:5101/admin).
280+
- Open `/admin` on the node's discovered HTTPS URL.
280281
- Or check the cluster from the CLI:
281282

282283
```bash
283-
dotnet run --project src/KeyLoad.Cli --no-build -- status http://localhost:5101
284+
dotnet run --project src/KeyLoad.Cli --no-build -- status "$KEYLOAD_ENDPOINT"
284285
```
285286

286287
## Use it
@@ -295,7 +296,8 @@ using KeyLoad.Client;
295296

296297
var apiKey = Environment.GetEnvironmentVariable("KEYLOAD_API_KEY")
297298
?? throw new InvalidOperationException("Set KEYLOAD_API_KEY.");
298-
using var http = new HttpClient { BaseAddress = new Uri("http://localhost:5101") };
299+
using var http = new HttpClient { BaseAddress = new Uri(Environment.GetEnvironmentVariable("KEYLOAD_ENDPOINT")
300+
?? throw new InvalidOperationException("Set KEYLOAD_ENDPOINT to the Aspire HTTPS URL.")) };
299301
var client = new KeyLoadClient(http, apiKey);
300302

301303
// tenant, database, transaction domain, partition key
@@ -351,7 +353,7 @@ Every node serves [MCP](https://modelcontextprotocol.io/) at `/mcp`. Its compact
351353
"mcpServers": {
352354
"keyload": {
353355
"type": "http",
354-
"url": "http://localhost:5101/mcp",
356+
"url": "${KEYLOAD_ENDPOINT}/mcp",
355357
"headers": { "Authorization": "Bearer ${KEYLOAD_API_KEY}" }
356358
}
357359
}

‎build/KeyLoad.AppHost/ClientApi/Resources/ClusterConnectionEndpointResources.cs‎

Lines changed: 0 additions & 18 deletions
This file was deleted.

‎build/KeyLoad.AppHost/Program.cs‎

Lines changed: 8 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,9 @@
11
using System.Globalization;
2-
using System.Net;
32
using System.Security.Cryptography;
43
using KeyLoad.AppHost.StorageRecovery;
54
using KeyLoad.AppHost.TestInfrastructure;
65
using KeyLoad.Clustering;
76
using Microsoft.Extensions.DependencyInjection;
8-
using Microsoft.Extensions.Options;
97

108
namespace KeyLoad.AppHost.Hosting;
119

@@ -23,7 +21,7 @@ internal static class AppHostProgram
2321
private const string StorageModeEnvironment = "KeyLoad__StorageMode";
2422
private const string SharedDirectory = "shared";
2523
private const string SiloNameEnvironment = "KeyLoad__SiloName";
26-
private const string HttpEndpoint = "http";
24+
private const string HttpsEndpoint = "https";
2725
private const string SiloEndpoint = "silo";
2826
private const string TcpScheme = "tcp";
2927
private const string ReadyPath = "/health/ready";
@@ -41,7 +39,6 @@ internal static class AppHostProgram
4139
private const string ServerUrlsEnvironment = "ASPNETCORE_URLS";
4240
private const string LoopbackAddress = "127.0.0.1";
4341
private const int FirstNodeIndex = 0;
44-
private const int LastNodeOffset = 1;
4542
private const int SharedVolumeCopies = 1;
4643

4744
private static async Task<int> Main(string[] args)
@@ -86,11 +83,6 @@ internal static IResourceBuilder<ProjectResource>[] AddCluster(IDistributedAppli
8683
var nodeNames = ClusterTopologyNodeNames.Read(benchmarkNodeCount, functionalNodeCount, ephemeral,
8784
options.Cluster.Value.SiloCount);
8885
var replicas = benchmarkNodeCount ?? options.Cluster.Value.ReplicaCount;
89-
if (!ephemeral && (long)options.Cluster.Value.FirstPublicPort + nodeNames.Length - LastNodeOffset > IPEndPoint.MaxPort)
90-
{
91-
throw new OptionsValidationException(Options.DefaultName, typeof(ClusterDeploymentOptions),
92-
[ClusterDeploymentOptions.ValidationMessage]);
93-
}
9486
if (replicas > nodeNames.Length
9587
|| (options.Cluster.Value.StorageMode == StorageAccessMode.SharedVolume && replicas != SharedVolumeCopies))
9688
{
@@ -105,11 +97,11 @@ internal static IResourceBuilder<ProjectResource>[] AddCluster(IDistributedAppli
10597
for (var index = FirstNodeIndex; index < nodes.Length; index++)
10698
{
10799
nodes[index] = AddNode(builder, databaseIdentity, root, ephemeral, nodeNames.Length,
108-
replicas, identity, nodeNames[index], index);
100+
replicas, identity, nodeNames[index]);
109101
}
110102
var contenders = ExclusiveOwnerContentionResources.Add(options.Control.Value.ExclusiveOwnerContention,
111103
nodes[FirstNodeIndex], resourceName => AddNode(builder, databaseIdentity, root, true,
112-
nodeNames.Length, replicas, identity, ExclusiveOwnerContentionResources.OwnerName, FirstNodeIndex,
104+
nodeNames.Length, replicas, identity, ExclusiveOwnerContentionResources.OwnerName,
113105
resourceName));
114106
var membershipSecret = builder.AddParameter(MembershipParameter,
115107
Convert.ToBase64String(RandomNumberGenerator.GetBytes(MembershipSecretBytes)), secret: true);
@@ -130,19 +122,18 @@ private static IResourceBuilder<ProjectResource> AddNode(IDistributedApplication
130122
ClusterIdentity databaseIdentity, string root, bool ephemeral, int siloCount, int replicaCount,
131123
(IResourceBuilder<ParameterResource> Signing, IResourceBuilder<ParameterResource> Admin,
132124
IResourceBuilder<ParameterResource> Incarnation) identity,
133-
string ownerName, int index, string? resourceName = null)
125+
string ownerName, string? resourceName = null)
134126
{
135127
resourceName ??= ownerName;
136128
var options = AppHostOptionsRegistration.Get(builder).Cluster.Value;
137129
var directory = Path.Combine(root, options.StorageMode == StorageAccessMode.SharedVolume ? SharedDirectory : ownerName);
138130
Directory.CreateDirectory(directory);
139-
var port = ephemeral ? (int?)null : options.FirstPublicPort + index;
140131
var resource = builder.AddProject<Projects.KeyLoad>(resourceName, launchProfileName: null)
141-
.WithHttpEndpoint(port: port, name: HttpEndpoint, isProxied: false)
132+
.WithHttpsEndpoint(name: HttpsEndpoint, isProxied: false)
142133
.WithEndpoint(name: SiloEndpoint, scheme: TcpScheme, isExternal: false, isProxied: false);
143134
resource.WithEnvironment(ClusterEnvironment, ClusterPrefix + databaseIdentity.Incarnation.ToString(ClusterGuidFormat))
144-
.WithEnvironment(PublicEnvironment, resource.GetEndpoint(HttpEndpoint))
145-
.WithEnvironment(ServerUrlsEnvironment, resource.GetEndpoint(HttpEndpoint))
135+
.WithEnvironment(PublicEnvironment, resource.GetEndpoint(HttpsEndpoint))
136+
.WithEnvironment(ServerUrlsEnvironment, resource.GetEndpoint(HttpsEndpoint))
146137
.WithEnvironment(SiloNameEnvironment, resourceName)
147138
.WithEnvironment(SiloCountEnvironment, siloCount.ToString(CultureInfo.InvariantCulture))
148139
.WithEnvironment(ReplicaCountEnvironment, replicaCount.ToString(CultureInfo.InvariantCulture))
@@ -155,8 +146,7 @@ private static IResourceBuilder<ProjectResource> AddNode(IDistributedApplication
155146
.WithEnvironment(IncarnationEnvironment, identity.Incarnation)
156147
.WithEnvironment(SigningEnvironment, identity.Signing)
157148
.WithEnvironment(AdminEnvironment, identity.Admin)
158-
.WithHttpHealthCheck(ReadyPath, endpointName: HttpEndpoint);
159-
KeyLoad.AppHost.ClientApi.ClusterConnectionEndpointResources.Apply(resource);
149+
.WithHttpHealthCheck(ReadyPath, endpointName: HttpsEndpoint);
160150
ClusterResourceSettings.Apply(builder, resource);
161151
ProcessKillCommand.Apply(builder, resource, ephemeral);
162152
return resource;
Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
using System.Net;
21
using KeyLoad.Clustering;
32
using Microsoft.Extensions.Configuration;
43

@@ -9,11 +8,9 @@ namespace KeyLoad.AppHost.TestInfrastructure;
98
internal sealed record ClusterDeploymentOptions
109
{
1110
private const int DefaultSiloCount = 3;
12-
private const int DefaultFirstPublicPort = 5101;
1311
private const string SiloCountKey = "KeyLoad:SiloCount";
1412
private const string ReplicaCountKey = "KeyLoad:ReplicaCount";
1513
private const string StorageModeKey = "KeyLoad:StorageMode";
16-
internal const string FirstPublicPortKey = "KeyLoad:FirstPublicPort";
1714
private const string LoggingKey = "Logging:LogLevel:Default";
1815
private const string DefaultLoggingLevel = "Warning";
1916
internal const string ValidationMessage = "Cluster deployment settings are invalid.";
@@ -26,15 +23,11 @@ internal sealed record ClusterDeploymentOptions
2623
/// <summary>Whether nodes receive isolated directories or the same shared directory.</summary>
2724
[ConfigurationKeyName(StorageModeKey)]
2825
public StorageAccessMode StorageMode { get; init; } = StorageAccessMode.IsolatedVolumes;
29-
[ConfigurationKeyName(FirstPublicPortKey)]
30-
public int FirstPublicPort { get; init; } = DefaultFirstPublicPort;
3126
[ConfigurationKeyName(LoggingKey)]
3227
public string LoggingLevel { get; init; } = DefaultLoggingLevel;
3328
internal bool IsValid() => SiloCount >= 1
3429
&& ReplicaCount >= 1 && ReplicaCount <= SiloCount
3530
&& Enum.IsDefined(StorageMode)
3631
&& (StorageMode != StorageAccessMode.SharedVolume || ReplicaCount == 1)
37-
&& !string.IsNullOrWhiteSpace(LoggingLevel)
38-
&& FirstPublicPort >= 1
39-
&& FirstPublicPort <= IPEndPoint.MaxPort;
32+
&& !string.IsNullOrWhiteSpace(LoggingLevel);
4033
}

‎docs/ADR/ADR-113-centralized-runtime-options.md‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -113,15 +113,12 @@ ceilings and rejects different policies within a comparable cohort.
113113
This metadata schema change does not change persisted database formats or qualify
114114
new performance measurements.
115115

116-
cluster published host ports consume `ClusterDeploymentOptions.FirstPublicPort`,
117-
centrally bound from `KeyLoad:FirstPublicPort` with the preserved default5101.
118-
Validate the complete range for the configured node count against execution `IPEndPoint` bounds
119-
before profile-directory or Aspire-resource creation. Ordinary composition uses
120-
the captured value plus node index; ephemeral composition still delegates port
121-
assignment to Aspire. Internal container HTTP/silo ports remain frozen image
122-
identities. `ClusterPublishedPortPolicyTests` verifies binding, every
123-
actual endpoint annotation, snapshot lifetime, the inclusive upper range and
124-
rejection before resource/file ownership (AC-CQ-034/035).
116+
Cluster HTTPS, silo and membership endpoints receive ports allocated by Aspire.
117+
KeyLoad does not select a first public port or maintain a port range.
118+
`ClusterPublishedPortPolicyTests` checks both ordinary and ephemeral project
119+
composition: the sole HTTPS endpoint and every Orleans endpoint leave host and
120+
target ports unset before Aspire allocation (AC-CQ-034/035). Container target
121+
ports belong to their separate published-image contract.
125122

126123
Verification order is a real full Release build, scoped formatting and a
127124
final full build, then Aspire-owned analyzers/unit/scalar/recovery/cluster plus the

‎docs/ADR/ADR-125-connection-owned-execution.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ flowchart LR
2424
- Lifetime. Disconnect uses Kestrel `ConnectionClosed`; `RequestAborted` belongs to one operation. Host shutdown closes connection admission and joins connections before the silo and storage stop. Reconnect obtains a new grain. Silo loss discards the disposable grain; command-receipt retry and uncertain writes keep their existing semantics.
2525
- Background work (timers, durable jobs, provider work) uses one bounded execution owner per silo service, never a grain per operation.
2626
- Errors. `KeyLoadException` is serialized ly (`[GenerateSerializer]`, alias `keyload.error`, Code id 0, StatusCode id 1) so domain codes survive Orleans RPC.
27-
- Transport. HTTP/1.1 runs sequential requests per connection. Overlapping calls on one connection use the dedicated HTTP/2 listener (`ServerExecutionOptions.EnableHttp2`, `Http2Port` default 8081); the AppHost exposes it on every node and clients use the Aspire-discovered endpoint. The SDK copies the supplied `HttpClient` request version and version policy into every request.
27+
- Transport. The AppHost exposes one HTTPS endpoint per node and Aspire allocates every public and Orleans port. TLS negotiates HTTP/1.1 or HTTP/2 on that endpoint; overlapping calls on one connection use HTTP/2. Clients use the Aspire-discovered HTTPS address. There is no separate cleartext HTTP/2 listener or application-selected port range. The SDK copies the supplied `HttpClient` request version and version policy into every request.
2828
- Current MCP transport is stateless; this decision does not deliver SQL session interoperability.
2929

3030
## Verification

‎docs/Features/QueryExecution/SignedCursorRequestIdentity.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ For EACH SDK and official MCP changed-OFFSET refusal, assert the exact CursorExp
1515
Replay the complete original eight-mutation seed command through EACH public client; compare every field of its original receipt and immediately verify full current state and fresh signed pages before another replay. No raw store/journal invariance or physical-cut/logical-commit inequality is claimed.
1616

1717
## AC-CURSOR-IDENTITY-004
18-
Join all original/new caller owners before EACH of two genuine all-three Aspire cluster cold cycles. Reopen fresh callers, reject old owner/read-generation cursors, verify fresh full literal pages and all original receipt replays. The unchanged original flow already owns two cold cycles: the strengthened complete Case therefore adds two, totaling four. Its original McpCallerDeadline and Test/Args identity are preserved.
18+
Join all original/new caller owners before EACH of two genuine all-three Aspire cluster cold cycles. Reopen fresh callers, repeat the changed-OFFSET refusal for the original signed cursors, verify fresh full literal pages and all original receipt replays. A same-volume restart preserves the physical owner identity and read generation unless storage replacement changes it; restart alone does not imply cursor expiry. The unchanged original flow already owns two cold cycles: the strengthened complete Case therefore adds two, totaling four. Its original McpCallerDeadline and Test/Args identity are preserved.
1919

2020
```mermaid
2121
flowchart LR
@@ -25,7 +25,7 @@ flowchart LR
2525
X --> V[Original OFFSET1 continues c then d]
2626
V --> R[Full seed receipt replay and current state]
2727
R --> K1[Join callers and all-three cold1]
28-
K1 --> F1[Old cursor refusal and fresh pages]
28+
K1 --> F1[Changed-OFFSET refusal and fresh pages]
2929
F1 --> K2[Join callers and all-three cold2]
3030
K2 --> F2[Fresh pages and original receipts]
3131
```

‎docs/Features/TestInfrastructure.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -322,9 +322,12 @@ REQ/AC-TEST-001: the full Unit model/disposal flow asserts every Aspire-owned en
322322

323323
REQ-TEST-001/011/015 and their existing ACs preserve the genuine configured-node model construction, complete endpoint assertions and joined model disposal. In ExactSelectionBuildsTheOwnedClusterModelAndSettlesDisposal the pre-DCP http2.Port is 8081, not null. No deployment port configuration changes are authorized by this test correction.
324324

325-
Pinned Aspire.Hosting 13.6.1's EndpointAnnotation retains an omitted backing port and exposes targetPort as Port for an unproxied endpoint unless the port was explicitly set to null. Existing ClusterConnectionEndpointResources.Apply supplies targetPort8081, name http2, isProxiedfalse and no port argument through WithHttpEndpoint. Therefore the constructed model's public Port property is8081; this is not proof of an explicitly fixed container host binding.
326-
327-
The pinned DcpExecutor allocation checks container SpecifiedPort for a fixed public port. The omitted backing port requires allocation for a proxyless container; EnsureProxylessEndpointPort assigns the real allocated public port while retaining container target8081. The cluster resources omit this http2 backing port. Model verification thus compares http2.Port to its independent Http2TargetPort constant before DCP, preserving exact names/order/scheme/target/internal/unproxied assertions, all other endpoint values, actual Aspire Build and original application/root cleanup. It does not claim actual distinct allocated endpoints, readiness, successful connection or cluster execution; those remain authentic Aspire runtime gates.
325+
The primary project AppHost uses one HTTPS endpoint per node. Aspire allocates
326+
its port and the silo/membership ports in ordinary and ephemeral composition.
327+
HTTP/1.1 and HTTP/2 share TLS negotiation on that endpoint. Integration clients,
328+
readiness checks and the CLI use the discovered HTTPS address. CI establishes
329+
trust in the .NET development certificate before starting the real topology;
330+
certificate verification stays enabled.
328331

329332
### Cold restart of every stopped node before readiness
330333

0 commit comments

Comments
 (0)