Dedicated rollup writer threads, a storage format reference, and rollup code cleanup.

[koculu]

Summary

With several ingestion workers, every worker wrote its own copy of each rollup row it had touched. This PR adds dedicated rollup writer threads, set by a new option, Ingestion.RollupWriterCount.

Each writer owns a share of the rollup rows. On flush, it merges what every worker buffered for its rows and writes each row once, instead of once per worker. The writers rely on the per-row lock stripes that are already on the branch.

The PR also adds a storage format reference page and removes duplicated rollup code in queries, backfill, retention and the key codecs.

Why

Before this change, each rollup row was written 5–13 times per 2-second flush window: once by every worker that had touched it.

Measured in the playground with 1M events and 20 workers:

Write path	Events/s	With rollup indexes
Lock stripes only (base)	45k	32k
4 rollup writers	55k	36k
8 rollup writers	60k	—

With 4 writers, CPU time per event is about 40% lower than in the base.

Changes

Rollup writer pool

• RollupWriterPool (new): dedicated writer threads, each owning rows by RollupBufferKey hash.
• Flushing:
  • A worker splits its buffer by writer, posts one piece to each writer and waits for all of them.
  • With one writer and nothing kept, the buffer is handed over without being split.
• Failure handling:
  • When a write fails, the writer keeps the batch and retries it before anything new.
  • While the retry keeps failing, the writer rejects new pieces. The worker keeps the rejected rows and counts them in RetainedRollupRows, so the existing back-pressure and flush recovery still apply.
  • The first failure is rethrown to the worker. Any other, different failures are logged.
• IngestionWorker:
  • Its own rollup write and retry state are removed; it now hands rows to the pool.
  • The flush order is unchanged: session deltas, then rollups, then session snapshots.
• AnalyticsEngine: creates the pool and owns its lifetime. Disposing the pool stops and joins its threads.
• What changes for callers:
  • if writing rollups fails, Flush, FlushAsync and Dispose now throw AnalyticsStorageException instead of the raw storage exception, such as IOException. Failures outside the rollup writers, such as session persistence, still throw the raw exception.

Options

• Ingestion.RollupWriterCount takes values from 1 to 64, and other values are rejected at validation. The default is 1, which is safe on single-processor machines.
• The dashboard sets RollupWriterCount = WorkerCount for both sample and benchmark ingestion.

Duplication cleanup (no change in behavior)

• RawSessionContributions: one definition of session contributions (bounces, duration, session page views, entrance, exit). Raw-session queries and rollup-index backfill both use it.
• RetentionCutoffs.RollupCutoff(granularity): the rule that day rollups use the local day and the others use UTC now lives in one place.
• ExpiredRecords:
  • IsRangeExpired and DeleteFirstOfExpiredHeads now take a KeyHeadLength function and work for any key layout.
  • They replace the separate copies for dimension indexes and rollup indexes.
  • AfterPrefix moved to KeyCodec.
• KeyCodec.ReadDimensionSetId: replaces inline segment parsing in RollupIndexStore, and a full DecodeRollup per row in RollupIndexBackfill.DeleteDefinitions.
• RollupAccumulator.MergeInto: one merge-or-add step with a single dictionary lookup. The pool, the writers and QueryEngine.MergeGroups all use it.
• RollupValueSerializer: the section reader and the entry reader are merged into one ReadLengthPrefixed. Malformed values now share one error message: "A rollup value section or entry is empty or truncated."
• What changes for callers:
  • If writing rollups fails, Flush, FlushAsync and Dispose now throw AnalyticsStorageException instead of the raw storage exception, such as IOException. Failures outside the rollup writers, such as session persistence, still throw the raw exception.

Docs

• New page: Storage format reference. It covers:
  • directory layout and partition routing;
  • key and value encodings;
  • sketches, sessions and indexes;
  • metadata generations;
  • recovery, expiration, and the rollup writer hand-off.
• Links: the new page is linked from the README, the reference index and the engine reference.
• Option docs: RollupWriterCount is added to the configuration and tuning pages.

Tests

• Writer pool tests (new):
  • AWriterWritesTheRowOfWorkersThatFlushedTogetherOnce
  • ManyWorkersAndWritersCountEveryEventOnce
  • FailuresOfSeveralWritersAreAllLogged
  • RollupWriterCountIsValidated
• Session metrics test (new): RebuiltRollupsRawQueriesAndLiveIngestionAgreeOnSessionMetrics checks that rebuilt rollups, raw queries and live ingestion give the same session metrics. It guards the shared session contributions.
• Adjusted tests:
  • The flush recovery harness now creates its own writer pool.
  • The persistent-failure test runs with one writer, because it counts flush attempts.
  • Dashboard option tests cover the writer counts.
  • Retention tests use the new head-length functions.