diff --git a/docs/README.md b/docs/README.md index f421eb3b1c..92873ded05 100644 --- a/docs/README.md +++ b/docs/README.md @@ -32,6 +32,7 @@ This section points to sources that explain why ServiceControl is designed the w - [Handling unavailable runtime dependencies](handling-unavailable-runtime-dependencies.md) — how instances react when a dependency is unavailable - [Telemetry](telemetry.md) — telemetry configuration and emitted metrics - [Throughput collection](throughput-collection.md) — why and how usage data is collected +- [Primary instance scale-out](primary-instance-scale-out.md): which parts of the primary may run as several processes, which must run once, and what a failover of the full instance needs ## Decisions and rationale diff --git a/docs/decisions/2026-10-09-primary-instance-scale-out.md b/docs/decisions/2026-10-09-primary-instance-scale-out.md new file mode 100644 index 0000000000..6df815fc98 --- /dev/null +++ b/docs/decisions/2026-10-09-primary-instance-scale-out.md @@ -0,0 +1,54 @@ +# Scale-out of the primary instance is limited to error ingestion + +- Date: 2026-10-09 +- Status: Accepted +- Implementation: [#5801](https://github.com/Particular/ServiceControl/pull/5801) added the `--error-ingestion-only` host and the single-owner split it relies on. Link further pull requests here as they open. + +## Context + +The primary instance is one process that does everything the error side of the platform needs. It ingests failed messages, stages and forwards retries, archives groups, monitors heartbeats and custom checks, dispatches integration events, collects usage data, sweeps retention and serves the HTTP API that ServicePulse polls. Its default storage is an embedded RavenDB server inside the process, so a second process sharing the data was never part of the design. + +SQL Server and PostgreSQL storage change that. Customers run those databases as highly available clusters, and once the database survives a node failure the question of whether the primary can do the same follows. It comes in two forms: several primaries behind a load balancer sharing the work, and a standby that takes over when the active one fails. The error ingestion path was built for concurrent writers from the start ([ingestion-pipeline.md](../ingestion-pipeline.md)), and #5801 used that to add ingestion workers that run beside one full instance. That change listed what a worker must not run: the retry pipeline, the retention sweep, integration event dispatch and heartbeat monitoring. It did not say whether the full instance itself could run more than once. + +An audit on 7 October 2026 read every hosted service, message handler, controller and persister path in the primary against both forms, with every full instance configured identically and the HTTP API behind a round-robin load balancer. [primary-instance-scale-out.md](../primary-instance-scale-out.md) holds the result per component. In summary: + +- Ingestion, retry acknowledgements, retry claims, every database-backed API read and write, ETags, authentication and scatter-gather already tolerate several processes. +- The retry pipeline assumes one process in five places. The staging batch is selected without a claim. Every node holds a receiver on the one staging queue, and the counting-mode forwarder can only finish when its own node has received the whole batch. The forwarding pointer is a single unguarded row. Orphan adoption treats every other process as dead. Retry and archive progress, and the guards between them, live in memory. Two full instances duplicate retries, then stop retrying altogether until a restart. +- Heartbeats are commands on the shared input queue, each node records the share it receives in memory, and each node runs the dead check. Two full instances report healthy endpoints as dead and restored, and publish those transitions to the event log, to integration subscribers and by email. +- Integration event dispatch is select, publish, delete with no claim, so every event reaches subscribers once per node. +- Throughput collection reads the last collected day and then adds absolute daily totals, and on RabbitMQ, SQL Server and PostgreSQL transports it samples a broker counter per process, so usage figures are multiplied by the node count. +- Several answers come from per-process state: the endpoints and heartbeat views, group retry and archive progress, license status, the MassTransit connector heartbeat and the manual purge status. +- Some state is local to the machine: file system message bodies, the license file, the NServiceBus host id that keys internal custom check rows, and MSMQ queues. + +Three constraints shape the decision: + +- The bulk retry design depends on one process serialising the staging and forwarding steps ([bulk-retries-design.md](../bulk-retries-design.md)). Making that safe across processes needs a lease or a claim at every step, and a defined handover for a lease holder that dies. +- Heartbeat readings arrive every ten seconds per endpoint instance and retry progress changes per batch, which is why both are kept in memory. Persisting them on every change adds write load and needs a schema the current persisters do not have. +- No test runs two full instances against one database. The existing concurrency tests exercise several writers inside one process, and the ingestion-only test asserts the hosted service set of a worker. + +## Decision + +Exactly one full primary instance runs against a database and its queues. Horizontal scale comes from `--error-ingestion-only` workers on SQL Server and PostgreSQL. High availability for the full instance is active-passive: a standby starts only after the active process has exited, and the two never run at the same time. + +The components that assume one process, and what each would need to become deployment-global, are listed in [primary-instance-scale-out.md](../primary-instance-scale-out.md). That list is the contract. A change that adds a hosted service, a message handler or a controller to the primary decides whether it may run on every node. A job that may run only once is switched off in `ErrorIngestionOnlyCommand` and added to the list, and `When_hosting_error_ingestion_only` fails when a hosted service appears that it does not expect. + +Nothing detects a second full instance at runtime, and this decision does not add detection. The constraint is stated in `Help.txt` and on the topic page. + +## Consequences + +- Customers get high availability by making the database highly available and running the full instance as a single-replica resource that an orchestrator or a cluster restarts or moves. A rolling update, or a cluster that starts the standby before the active has stopped, produces the duplicate retries and false heartbeat alarms the audit describes. Accepted and documented. The recovery paths a standby relies on (orphan adoption, forwarding resume and at-least-once dispatch) already exist for a single-node restart. +- A standby start replays single-node restart behaviour. Every monitored endpoint shows as failing until its next heartbeat, monitoring is re-enabled for endpoints a user disabled, up to one batch of integration events is published again, and a forwarding batch completes 45 seconds after the staging queue goes quiet. Accepted. The first two are what a restart does today. +- Everything except ingestion is bounded by one process. Accepted. Ingestion is the path whose load grows with the system's message volume, and the workers cover it. +- The HTTP API has one target. A load balancer in front of the primary only provides failover, because workers host only the health endpoints. Accepted. +- File system body storage must be a shared mount whenever more than one host writes or reads bodies. This is not enforced at startup. Accepted for now, with the startup warning that #5801 left open as follow-up work. +- An operator who starts two full instances by mistake gets no error, only the symptoms on the topic page. The risk is bounded by the symptoms being loud: retries stop and the event log fills with heartbeat alarms. A startup lease would turn the mistake into a refusal. See the alternatives. +- The per-component list ages with the code. Mitigation: the review questions on the topic page, and the ingestion-only test as the tripwire for hosted services. + +## Alternative approaches + +- Database leases for the jobs that may run once. A lease row or an advisory lock (`sp_getapplock`, `pg_advisory_lock`) around the retry processor and staging receiver, the retention sweep, the dispatcher, the throughput collectors and the settings sync would let several full instances run with one of them doing the single-owner work. This is the main point of leverage and the first step if active-active is ever offered. It is deferred because a lease on its own leaves the in-memory API state, the heartbeat split and the per-node license as they are, and because every lease needs renewal, expiry and a defined handover for a holder that dies, which is a redesign of the retry recovery paths. +- Persist the in-memory state. A last-seen time per endpoint instance written on every heartbeat, retry and archive progress written per batch, the connector heartbeat and the purge status in rows, and the groups and endpoints API served from the database. Together with leases this is the full active-active design. It is deferred for the same reason, and because it only pays for itself once leases exist. +- A designated owner node. A per-node setting, like `RunRetentionSweep` today, that turns the single-owner jobs off on all but one full instance, so the others serve the API and ingest. It is rejected because heartbeats and commands still arrive on the shared input queue and are consumed by every node, so the owner's view of endpoints and retry progress stays partial, and because a second owner is a configuration error nothing detects. +- Separate queues per concern. Route heartbeats, custom check reports and retry commands to a queue only the owner node consumes. It is rejected because it changes the platform connection contract every endpoint and ServicePulse rely on, and it still needs the owner designation above. +- Sticky sessions at the load balancer. They are rejected because they make the in-memory API views consistent for one client and change nothing about the background jobs or the heartbeat split. +- Transport-level exclusivity, such as a single active consumer on RabbitMQ. It is rejected because it is not available on every supported transport, and because it covers only the receivers and none of the timers. diff --git a/docs/primary-instance-scale-out.md b/docs/primary-instance-scale-out.md new file mode 100644 index 0000000000..d1bc4edaef --- /dev/null +++ b/docs/primary-instance-scale-out.md @@ -0,0 +1,129 @@ +# Primary instance scale-out + +The primary instance ingests failed messages, runs retries and archiving, monitors heartbeats and custom checks, dispatches integration events, collects usage data and serves the HTTP API that ServicePulse polls. This page says how many primary processes may run against one database and one set of queues, what each supported arrangement runs, and which parts of the full instance assume that they are the only process. The reasoning is in [the scale-out decision](decisions/2026-10-09-primary-instance-scale-out.md). How ingestion batches are written in parallel is covered in [ingestion-pipeline.md](ingestion-pipeline.md), and the retry state machine in [bulk-retries-design.md](bulk-retries-design.md). + +## Supported arrangements + +### One full instance + +The default. Every component runs in one process, and the in-memory state listed below is complete because that process sees every message and every request. + +### One full instance plus error ingestion workers + +`ServiceControl.exe --error-ingestion-only` runs a host that drains the error queue into the shared database and nothing else. Any number of workers may run beside exactly one full instance. The mode requires SQL Server or PostgreSQL storage. It refuses to start on RavenDB, because the RavenDB persister merges failed messages with patch scripts that are not ordered against each other (`RavenIngestionUnitOfWorkFactory`). A worker hosts the event log, external integration writes, recoverability (ingestion only), heartbeat monitoring (warm-up only, no checking) and custom checks components, and no NServiceBus endpoint. The acceptance test `When_hosting_error_ingestion_only` asserts the exact set of hosted services a worker runs, so adding a hosted service anywhere forces a decision about whether it may run on every node. + +Workers need: + +- A database already provisioned by the full instance's `--setup`. A worker runs no setup. +- Message bodies every host can read. File system body storage only works on a shared mount. Azure Blob and S3 storage work as they are. + +### Active-passive failover of the full instance + +A standby copy of the full instance may take over after the active one has stopped. The standby must start only after the active process has exited. The two must never run at the same time, because the components listed below behave as if they are alone. In practice that means a cluster resource or an orchestrator with a single replica and a recreate strategy, never a rolling update. + +Both copies need identical configuration: the instance name, which names the input queue; the hostname and port, which derive `Settings.InstanceId` for body URLs and the single-message retry route; the license, which is read from each machine; and the body storage location, which must be a shared mount, Azure Blob or S3. The transport must be a broker or a database. MSMQ and the Learning transport are machine-local, so the standby would own a different set of queues. Embedded RavenDB cannot fail over, because its data directory is local to the machine. Failover needs SQL Server, PostgreSQL or an external RavenDB server. + +A standby start replays what a single-node restart does today: + +- Retry batches the active node was still claiming (`MarkingDocuments`) are adopted and staged. A batch it was forwarding resumes in timeout mode and completes 45 seconds after the staging queue goes quiet (`RetryProcessor`, `RetryDocumentManager`). +- Endpoint monitoring is warmed from `KnownEndpoints` with no heartbeat readings, so every monitored endpoint counts as failing until its next heartbeat arrives, and the first heartbeat re-enables monitoring for endpoints a user had disabled (`EndpointInstanceMonitor`). +- Integration events the active node published but had not yet deleted are published again. Dispatch is at-least-once, and subscribers must tolerate duplicates (`ExternalIntegrationRequestsDataStore`). +- Internal custom check rows keyed on the old host id stay in the database with their last status. + +### Not supported: more than one full instance + +Two or more full instances against one database and one set of queues, whether deliberately behind a load balancer or through an overlapping failover, is not supported. Nothing refuses the second start. The sections below list what goes wrong and what each component would need. + +## What is safe to share today + +These paths already tolerate several processes, which is what the ingestion workers rely on: + +- Error ingestion. Competing consumers on the error queue, and a batch writer whose upserts are guarded by attempt times, whose inserts tolerate a competing writer's identical row, and whose lock order is stable (`FailedMessageBatchWriter` and the two `IFailedMessageIngestionSqlDialect` implementations). `ErrorIngestionConcurrencyTests` covers it. +- Retry acknowledgements and retry claims. A successful retry resolves the message only when no newer attempt is stored, and claims are insert-if-absent, so two batches never stage the same message (`RetryBatchStore`, `IRetryBatchSqlDialect`). +- Every controller that reads from and writes to the database directly. The ETag is a digest of the returned rows and totals with no per-process input (`DataVersion`), so a conditional GET answered by another node works. +- Authentication. JWT bearer tokens are validated against the OpenID Connect authority on every request. There is no cookie, session or data protection key ring. +- Scatter-gather to audit instances, the health endpoints, CORS, forwarded headers and pagination hold no server state between requests. Readiness reports the node's own ingestion watchdog, which is what a load balancer probe needs. +- Subscription rows on SQL Server and PostgreSQL, settings rows, custom check rows and known endpoint rows are written through upserts that handle the insert race. +- Azure Blob and S3 body storage, and file system body storage on a shared mount. Bodies are immutable and keyed by message id. + +## What assumes one process + +Each table names the component, the state or job that belongs to one process, what a second full instance does to it, and what it would need to become deployment-global. File names are given so a row can be checked against the code. Line numbers are not, because they change. + +### Retries + +| Component | Per-process state or unguarded job | With a second full instance | What it would need | +| --- | --- | --- | --- | +| Staging pass (`RetryProcessor`, `RetryStagingStore`) | `GetStagingBatch` is a plain select of the oldest `Staging` batch, and nothing marks it taken until `MarkBatchAsForwarding` | Both nodes stage the same batch. Every message is sent to the staging queue twice under two staging ids, the event log and audit entries are duplicated, and the last `MarkBatchAsForwarding` wins | Claim the batch with a conditional status update before staging, or run the processor under a lease | +| Forwarding pass (`RetryProcessor`, `ReturnToSenderDequeuer`) | Every node holds a receiver on the one `{InstanceName}.staging` queue and follows the same forwarding pointer. In counting mode the forwarder exits only when its own counter reaches the batch size | Both nodes run the forwarder for the same batch, the transport splits the messages between them, neither reaches its count, and both retry processors block until a restart. Messages carrying the other staging id are consumed and dropped | One receiver per batch, and an inactivity timer in counting mode so a short count cannot block forever | +| Forwarding pointer (`RetryBatchNowForwardingEntity`, `RetryStagingStore.PointForwarderAt`) | A single row written update-else-insert with no concurrency token, deleted after every batch | Two nodes staging different batches either collide on the insert, an unhandled key violation, or the second overwrites the first and leaves a `Forwarding` batch nothing will ever forward | A guarded upsert, and a recovery pass for `Forwarding` batches without a pointer | +| Orphan adoption (`RetryDocumentManager`, `AdoptOrphanBatchesFromPreviousSessionHostedService`) | Orphans are `MarkingDocuments` batches whose `RetrySessionId` differs from this process's static GUID. The sweep runs at startup, repeats while it finds something, and stops for good when it finds nothing | A starting node adopts batches a live node is still claiming and stages them half-built. A node that crashes beside a running peer leaves batches no running node will ever adopt | A liveness record per session, or adoption under the same lease as the processor | +| Recovery flag (`RetryProcessor.isRecoveringFromPrematureShutdown`) | A per-process boolean, true until the node's first staging pass | A node restarted while its peer forwards treats the live batch as abandoned, forwards it in timeout mode and deletes it. The peer's counting forwarder never completes and consumes a share of every later batch | Record ownership on the pointer instead of inferring it from process state | +| Retry progress (`RetryingManager`, `InMemoryRetry`) | Operation state and progress in a per-process dictionary that is never persisted and never expires. Startup seeds a `Preparing` entry for every open batch group | Progress exists only on the node that processes the batches. Other nodes show the operation as preparing forever, silently drop `RetryAllInGroup` and bulk requests for it, and refuse to archive the group | Persist operation state and read it back in the API and the guards | +| Groups API (`GroupFetcher`, `FailureGroupsController`) | Merges database rows with `RetryingManager` and `OperationsManager` progress | Status, progress and acknowledgement flags differ per node. While a group batch forwards, nodes that never saw the operation throw in `MapOpenForForwardingOperation` and answer 500 | Same as above | +| Bulk requests (`RetriesGateway`, `BulkRetryBatchCreationHostedService`) | Requests queue in process on the node that handled the command, deduplicated against that node's operations | Two nodes run the same bulk retry. Claims keep each message to one batch, but the second operation completes with zero messages and writes its own history and acknowledgement rows | Deduplicate requests in the database | +| Archive progress and guards (`OperationsManager`, `EFCoreArchivingManager`, `ArchiveAllInGroupHandler`, `RetryAllInGroupHandler`, `UnacknowledgedGroupsController`) | Archive progress in per-process dictionaries. The archive-while-retrying and retry-while-archiving guards read them, and acknowledging a completed archive removes a local entry | The guards are bypassed across nodes, so an archive and a retry of one group interleave and archived messages are forwarded. The completed-archive card appears on one node, and the acknowledge call answers 404 elsewhere | Persist progress (the `ArchiveOperations` row already carries counts) and base both guards on database state | +| Error queue address (`ErrorQueueNameCache`, `ReturnToSenderDequeuer`, `EditHandler`) | Set by the dequeuer's `StartAsync`, which runs after the NServiceBus endpoint has started receiving | A node that handles `EditAndSend` in that window has already marked the original resolved and then throws. The command ends in the instance's own error queue | Resolve the address at composition time | + +### Heartbeats and endpoint monitoring + +| Component | Per-process state or unguarded job | With a second full instance | What it would need | +| --- | --- | --- | --- | +| Heartbeat recording and the dead check (`HeartbeatHandler`, `EndpointInstanceMonitoring`, `HeartbeatMonitoringHostedService`) | Heartbeats are commands on the shared input queue. Each node records the share it received in a per-process dictionary and runs the check every 5 seconds against the 40 second grace period. No last-seen time is persisted | Each node sees a fraction of every endpoint's heartbeats, declares healthy endpoints dead and restored, and writes the event log rows, integration events and emails for each transition | Persist a last-seen time per endpoint instance and run the check on one node | +| Monitoring API (`EndpointsMonitoringController`) | `GET /api/endpoints`, `/api/endpoints/known` and `/api/heartbeats/stats` answer from the dictionaries, which are loaded from `KnownEndpoints` once at startup | Lists, counts, last report times and ETags differ per node. Endpoints detected on one node are absent on the others until they restart | Serve the endpoints from the database | +| Monitoring toggles and deletes (`EndpointsMonitoringController`, `MonitoringDataPersister`) | `PATCH` and `DELETE /api/endpoints/{id}` check existence in memory and mutate the serving node's monitor plus the database row | Other nodes keep the old flag and the deleted endpoint, keep raising heartbeat events for it, and answer 404 for an endpoint they never saw | Check existence and read the flag from the database at decision time | +| First heartbeat (`EndpointInstanceMonitor`, `MonitoringDataStore.CreateOrUpdate`) | The first heartbeat a node sees for an instance sets `Monitored` in memory and writes that value to the row | Every node start re-enables monitoring for endpoints a user disabled, and writes a "confirmed to have heartbeats" event log row per node | Insert-if-missing, and never touch `Monitored` on an existing row | +| Instance purge (`HeartbeatEndpointSettingsSyncHostedService`) | Every 6 hours, for endpoints with instance tracking off, deletes `KnownEndpoints` rows for instances that this node's memory says are not sending heartbeats, keeping the last one in its own dictionary order. Then deletes settings rows for endpoint names with no known endpoint row | Live instances are deleted. Two nodes keep different survivors, so every row of an endpoint can go, and the following sweep removes the user's tracking setting and recreates it with the default | Run on one node from persisted liveness | +| MassTransit connector heartbeat (`MassTransitConnectorHeartbeatStatus`) | The last connector heartbeat in a per-process property, read by `/api/configuration` and the license page | Presence, queue list and the license extension link differ per node | Persist the latest heartbeat | + +### Integration events and subscriptions + +| Component | Per-process state or unguarded job | With a second full instance | What it would need | +| --- | --- | --- | --- | +| Dispatch on both persisters (`ExternalIntegrationRequestsDataStore`, `EventDispatcherHostedService`) | Select, publish, then delete, with a per-process semaphore as the only exclusion | Both nodes publish the same batch, so every contract event reaches subscribers up to once per node | Claim rows before publishing, with a delete that returns the rows or a select for update that skips locked rows, or one dispatcher under a lease | +| Subscription cache (`SubscriptionStorage`, SQL Server and PostgreSQL) | Subscriber lookups are cached per process for `SubscriptionCacheDuration`, 60 seconds by default, and a subscribe or unsubscribe invalidates only the handling node's cache | Other nodes publish to a stale subscriber list for up to the cache duration. Message-driven transports only | Set the duration to zero on multi-node deployments, or version the table | +| Subscription document (`RavenSubscriptionStorage`) | All subscriptions in one document, mirrored into memory at startup and stored back whole with no change vector | A node never learns about subscriptions made through another, and its next save overwrites the document with its stale copy | Reload per lookup and store with optimistic concurrency | + +### Throughput collection and licensing + +| Component | Per-process state or unguarded job | With a second full instance | What it would need | +| --- | --- | --- | --- | +| Audit and cloud broker collectors (`AuditThroughputCollectorHostedService`, `BrokerThroughputCollectorHostedService`) | Read `LastCollectedDate`, fetch absolute daily totals, then add them through the additive upsert (`IEndpointThroughputDialect`). Both start 40 seconds after host start, then daily | Overlapping gathers add the same days twice, so audit, Azure Service Bus and Amazon SQS throughput is multiplied for every day in the window | One collector under a lease, or an idempotent set for absolute totals | +| Sampling broker queries (`RabbitMQQuery`, `SqlServerQuery`, `PostgreSqlQuery`) | Sample a shared broker counter from a per-process baseline and record each delta | Every node adds its own deltas, so broker throughput is multiplied for as long as the nodes run | One collector under a lease | +| License state (`ActiveLicense`, `LicenseCheckHostedService`, `LicenseController`) | Read from each machine's license sources into a per-process singleton, refreshed every 8 hours or on `?refresh=true` for that node only | Nodes with different license files answer differently, and a refresh fixes one node | Install the same license on every machine and refresh each | + +### Retention + +| Component | Per-process state or unguarded job | With a second full instance | What it would need | +| --- | --- | --- | --- | +| Sweep (`RetentionSweeper`) | Registered on every full instance (`PersistenceFactory` sets `RunRetentionSweep` from the ingestion-only flag). The single-flight guard is a per-process semaphore | Every node sweeps hourly. The end state is correct because the deletes re-assert the cutoff, but the work is duplicated and the sweeps overlap on a backlog | A database lease, or a per-node setting | +| Manual purge (`SystemMaintenanceController`, `RetentionApi`) | Start, the 409 "already running" answer and the status come from the serving node's sweeper | A second node accepts a second purge, and a status poll routed to another node reports it finished | Persist the purge run | + +### Custom checks and notifications + +| Component | Per-process state or unguarded job | With a second full instance | What it would need | +| --- | --- | --- | --- | +| Internal checks (`InternalCustomChecksHostedService`, `CustomCheckDetail`) | Rows are keyed by instance name, host id and check id. The host id is NServiceBus's default, derived from the machine name and executable path, and nothing removes a dead host's rows | One row per node for every internal check, so a shared condition such as a failed import raises N failures, N emails and N integration events. Two nodes with the same host id flip one row. A replaced node's rows stay | Key deployment-global checks on the instance name only, and expire stale rows | +| Email throttling (`EmailThrottlingState`, `SendEmailNotificationHandler`) | The throttle after an SMTP failure and the send semaphore are per process | During an SMTP outage every node attempts and fails on its own | Acceptable, or a shared throttle row | +| Saga audit misconfiguration (`SagaAuditMisconfigurationCustomCheck`) | A static flag set by the handler on the node that consumed the misrouted message | Only that node's row can fail | Persist the detail | +| Remote availability (`CheckRemotes`, `RemoteInstanceSetting.TemporarilyUnavailable`, `ScatterGatherApi`) | A per-process flag that skips an audit instance for up to 30 seconds after a failure | Consecutive requests alternate between complete and incomplete results while an audit instance is unreachable from one node | Acceptable as transient | + +### Storage, transport and hosting + +| Component | Per-process state or unguarded job | With a second full instance | What it would need | +| --- | --- | --- | --- | +| File system body storage (`FileSystemBodyStoragePersistence`, `BodyStorage`) | Bodies are files under the configured path on the ingesting node | On other nodes the body is unavailable. The API answers 204, a retry forwarded by another node throws, and the sweep deletes the row while the file stays | A shared mount, Azure Blob or S3, and a startup warning when the path is local | +| MSMQ and Learning transports (`MsmqTransportCustomization`, `LearningTransportCustomization`) | Queues are local to the machine | Nodes on different machines own disjoint queues and never compete, and a failover strands the old machine's queues | Broker or database transports only | +| Embedded RavenDB (`RavenEmbeddedPersistenceLifecycle`) | An in-process server with a local data directory | Each node holds a disjoint database | External RavenDB, SQL Server or PostgreSQL | +| Import failed errors (`ImportFailedErrorsCommand`) | Hosts a full NServiceBus endpoint on the instance input queue with every handler registered but only the recoverability component configured | Heartbeats, custom check reports and endpoint registrations it dequeues fail activation and land in the error queue. A running instance must be stopped first, which a highly available deployment cannot do | A send-only host, or an in-process import | +| Instance id (`Settings.InstanceId`) | Derived from each node's hostname, port, virtual directory and TLS scheme | Body URLs and the single-message retry route carry the serving node's id, and another node answers 400 for an id it does not recognise, unless every node is configured with the load balancer's address | Identical hostname and port on every node | +| Setup (`SetupCommand`, `--setup-and-run`) | Runs migrations, queue creation and body storage provisioning per node | Migrations serialise on EF Core's migration lock. S3 bucket creation is check-then-create, and the loser's setup fails | Run `--setup` once, and start the others with plain run | + +Smaller effects: event log `RaisedAt` and retention cutoffs come from whichever node wrote or swept, so clock skew between nodes shifts ordering and expiry by the skew. The message action audit trail is written to each node's own log sink, so one operation is split across files and has to be joined on the operation id. The usage report's `Host.*` and `Heartbeats.*` environment values describe the node that built it. Retry history can briefly exceed the configured depth. Group archive batch events name the fetched ids rather than the affected rows. + +## Changing a component + +A new hosted service, message handler or controller in the primary has to answer two questions before it is merged: + +- Does it keep state that must be the same on every node, such as a dictionary that backs an API answer or a guard? If so, the state belongs in the database, or the component joins the tables above with what it would need. +- Is it a job that a deployment may run only once? If so, it is switched off in the ingestion-only host (`ErrorIngestionOnlyCommand`) and listed above. `When_hosting_error_ingestion_only` fails when a hosted service appears that the test does not expect.