Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Already cloned without it? `git lfs pull` fixes it in place. The `dev` dataset i
npm run check
```

That checks, each with a committed negative test, that `SPEC.md` agrees with the requirement registry, that every MUST is covered, that the promotion index returns a superset of the eligible set, that the dataset matches its manifest, that the tree typechecks, and that a simulated deploy lands nothing dev-only on a node.
That checks, each with a committed negative test, that `SPEC.md` agrees with the requirement registry, that every reference to a section of `SPEC.md` resolves to a heading that exists, that every MUST is covered, that the promotion index returns a superset of the eligible set, that the dataset matches its manifest, that the tree typechecks, and that a simulated deploy lands nothing dev-only on a node.

Then the conformance suite, which is the executable form of the specification:

Expand Down
77 changes: 53 additions & 24 deletions SPEC.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Run `bench/run.mjs` directly and you get none of the orchestrator's guarantees:

## Why the access distribution is load-bearing

Keys are drawn with skew because uniform draws make the cache unmeasurable. A uniform draw over the catalog crossed with tier and region spans a key space a short run almost never revisits, and the measured hit rate collapses to near zero. A cache nobody asks for twice cannot be measured, and neither can the cost of invalidating it — which silently takes `WRITE-003` and most of §6 out of reach without any test failing.
Keys are drawn with skew because uniform draws make the cache unmeasurable. A uniform draw over the catalog crossed with tier and region spans a key space a short run almost never revisits, and the measured hit rate collapses to near zero. A cache nobody asks for twice cannot be measured, and neither can the cost of invalidating it — which silently takes `WRITE-003` and most of the write-coherence requirements out of reach without any test failing.

The skew is defined in `@ecommerce-store/spec` and shared with the dataset generator, so the dataset's hot products and the workload's hot products are the same products. A hot subset is what makes cache-hit rate mean anything.

Expand Down
4 changes: 2 additions & 2 deletions bench/run.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ const OPS_URL = argOf('ops', 'http://localhost:9925');
*
* `harper dev` disables auth; `harper run` — the only supported benchmark configuration —
* does not. Without these every write 401s, `fetch` resolves normally, nothing checks the
* status, and the run record still reports `writesIssued`. The entire SPEC.md §6
* status, and the run record still reports `writesIssued`. The entire SPEC.md#background-writes
* cache-coherence workload silently does not happen.
*/
const OPS_AUTH = process.env.HDB_ADMIN_USERNAME
Expand Down Expand Up @@ -308,7 +308,7 @@ async function step({ rate, seconds, nextRequest, nextWrite, writeRate, label })
writesIssued,
writeFailures,
// Cache entries invalidated by this step's writes. The invalidation fan-out is the cost
// SPEC.md §6 exists to measure, so it is recorded rather than inferred.
// SPEC.md#background-writes exists to measure, so it is recorded rather than inferred.
invalidations,
// Requests still outstanding when the drain deadline expired. They are absent from the
// latency distribution, so a non-zero value means the tail is censored.
Expand Down
6 changes: 3 additions & 3 deletions bench/workload.mjs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/**
* The request mix — SPEC.md §4, §5, §6.
* The request mix — SPEC.md#post-cartidquote, "GET /product/:id", "Background writes".
*
* Deliberately shares nothing with the application. The harness must be able to drive any
* implementation of the specification, so it speaks HTTP and canonical paths only.
Expand Down Expand Up @@ -83,7 +83,7 @@ export function makeRequestFactory({ baseUrl, cartIds, productIds, quoteShare =
}

/**
* The background write stream — SPEC.md §6.
* The background write stream — SPEC.md#background-writes.
*
* Not an endpoint under test. It exists so caches have to stay coherent with their source
* of truth: a read-only workload lets a cache fill once and never invalidate, which is not
Expand All @@ -102,7 +102,7 @@ export function makeWriterFactory({ baseUrl, skus, inventoryIds, seed = 7 }) {
* Writes go through the APPLICATION, not the operations API.
*
* A direct table write updates the source of truth and invalidates nothing, so the cache
* would only converge on expiry and §6's cache-coherence cost — the thing it exists to
* would only converge on expiry and SPEC.md#background-writes's cache-coherence cost — the thing it exists to
* measure — would never be paid. A separated stack's write path has to evict its Redis key
* for the same reason; this is the same work on the other architecture.
*/
Expand Down
2 changes: 1 addition & 1 deletion containers/compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ services:
TARGET_URL: http://harper:9926
OPS_URL: http://harper:9925
# The background write stream authenticates against the operations API. Without these
# every write 401s, fetch resolves normally, and the SPEC.md §6 coherence workload
# every write 401s, fetch resolves normally, and the SPEC.md the "Background writes" coherence workload
# silently does not happen while the run record still reports writesIssued.
HDB_ADMIN_USERNAME: ${HDB_ADMIN_USERNAME:-benchadmin}
HDB_ADMIN_PASSWORD: ${HDB_ADMIN_PASSWORD:?set in containers/.env}
Expand Down
7 changes: 3 additions & 4 deletions docs/data-model.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Data model

Status: **P0 design.** Implements SPEC.md §3.
Status: **P0 design.** Implements [Data model](../SPEC.md#data-model).

SPEC.md defines the eight tables neutrally. This document records the Harper-side design and the reasoning behind the choices that are not forced.

Expand Down Expand Up @@ -108,8 +108,7 @@ type Rate @table @export @sealed {
}
```

And the one table the specification does not describe, because it is derived rather than a
source of truth:
And the one table the specification does not describe, because it is derived rather than a source of truth:

```graphql
# A cache, not an entity. CACHE-001: dropping it changes no response body, only latency.
Expand All @@ -134,5 +133,5 @@ type ProductView @table(expiration: 120) {

## Open

- **The promotion evaluation order in SPEC.md §4 is invented**, not derived from a real pricing engine. It is normative because the alternative is two correct implementations disagreeing on a total — but it should be sanity-checked against someone who has built one.
- **The promotion evaluation order in [POST /cart/:id/quote](../SPEC.md#post-cartidquote) is invented**, not derived from a real pricing engine. It is normative because the alternative is two correct implementations disagreeing on a total — but it should be sanity-checked against someone who has built one.
- **How selective the promotion index can get.** It narrows to 7.2% of the table on `dev`. Tier is the floor: four values means a tier probe can never return less than roughly a quarter plus the storewide share. A composite `(tierKey, categoryKey)` index, or folding tier into the category key, would go further — worth measuring at `bench` scale before building.
27 changes: 11 additions & 16 deletions docs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The benchmarks repo's methodology is binding here. Results are never published f

| Piece | State |
|---|---|
| `SPEC.md` | Written; two rounds of cross-model review applied. **Review gate not passed** — see SPEC.md §8, which is the only place gate status is tracked |
| `SPEC.md` | Written; two rounds of cross-model review applied. **Review gate not passed** — see [Conformance](../SPEC.md#conformance), which is the only place gate status is tracked |
| Schema | Built — the eight specified tables, plus `ProductView`, a derived cache |
| Caching | `ProductView`, a `sourcedFrom` cache keyed by product × tier × region, invalidated write-through with an expiry backstop |
| Write surface | `POST /admin/variant/:sku`, `POST /admin/inventory/:id` — write and invalidate |
Expand All @@ -38,25 +38,24 @@ The benchmarks repo's methodology is binding here. Results are never published f
## Next

1. **A `bench`-scale run that completes.** No run has yet finished the full ladder at `bench` scale, so the observations below are partial and nothing downstream of them is settled.
2. **A ground-truth correctness guard in the harness.** The measurement rules require it and the harness does not have it. This is the largest outstanding hole in the measurement story — see `bench/README.md`.
3. **Confirm `FRESH_MS`.** SPEC.md §6 now sets it provisionally; the figure wants a real run behind it, and `PDP-003` / `WRITE-002` depend on it.
2. **A correctness suite with expected values.** The measurement rules require a ground-truth guard and the harness does not have one — see `bench/README.md`. The shape: a suite separate from conformance, keyed to the loaded dataset's manifest checksum, asserting *values* rather than properties. Conformance asserts a quote is well-formed and self-consistent; two implementations can both pass it and disagree on every total, which is what this closes.

The expected values are **data, not a shared function**. A calculator in this repo emits them; no other implementation imports anything. A stack that computes pricing inside one SQL query compares against the same table as Harper does — the answer is stack-neutral, the procedure deliberately is not.
3. **Confirm `FRESH_MS`.** [Background writes](../SPEC.md#background-writes) now sets it provisionally; the figure wants a real run behind it, and `PDP-003` / `WRITE-002` depend on it.
4. **An in-application background writer.** The harness drives writes through the operations API. That is fine for coherence pressure but is not the same as the application doing it.

## Blocked on someone else

| | Needed from |
|---|---|
| **A Linux host.** On macOS, Podman runs in a VM: CPU/memory limits bind against the VM's share, not the host's, and the VM boundary sits in the network path. Clock pinning is impossible, so no cycle-normalized efficiency figure exists. | Maintainers |
| **Review gate** — tracked in SPEC.md §8. Human review outstanding. | Maintainers |
| **Review gate** — tracked in [Conformance](../SPEC.md#conformance). Human review outstanding. | Maintainers |
| **The promotion evaluation order is invented.** Normative because two correct implementations otherwise disagree on a total. Nobody who has built a pricing engine has read it. | A reviewer with pricing experience |
| **OAuth authorization-server question** — drafted at [`questions/oauth-authorization-server-scope.md`](questions/oauth-authorization-server-scope.md), unsent. Not blocking; auth is future work. | `@harperfast/oauth` maintainers |

## First bench-scale observations (2026-10-05) — NOT results

A `bench`-scale run reached three ladder steps before the load generator died of heap
exhaustion. The numbers below are recorded because they are the first evidence the dataset
is doing its job, and discarded as measurements because the run did not complete, the
generator was in trouble throughout, and the clock was not pinned.
A `bench`-scale run reached three ladder steps before the load generator died of heap exhaustion. The numbers below are recorded because they are the first evidence the dataset is doing its job, and discarded as measurements because the run did not complete, the generator was in trouble throughout, and the clock was not pinned.

| offered | achieved | p50 | p90 | p99 | errors | cache hit |
|---|---|---|---|---|---|---|
Expand All @@ -68,14 +67,9 @@ Cold start to first response: **3,056 ms** — the first cold number the project

What it suggests, pending a run that completes:

- **Capacity is somewhere between 100 and 200 rps**, against 2,000+ rps at `dev` scale on the
same container. A ~13x drop is what a working set exceeding memory looks like, so
`DATA-004` appears to be doing exactly what it was written for.
- **The ladder default brackets this range.** A ladder whose steps all sit past collapse
locates no inflection point, which is the one thing the ladder exists to find.
- **Cache hit rate climbs but stays low** (18% → 33%). At `dev` it reached 77% in the first
step. Worth watching: it may simply be a cold cache over a far larger key space, or the
hot set may be too diffuse at this catalog size even with the skew.
- **Capacity is somewhere between 100 and 200 rps**, against 2,000+ rps at `dev` scale on the same container. A ~13x drop is what a working set exceeding memory looks like, so `DATA-004` appears to be doing exactly what it was written for.
- **The ladder default brackets this range.** A ladder whose steps all sit past collapse locates no inflection point, which is the one thing the ladder exists to find.
- **Cache hit rate climbs but stays low** (18% → 33%). At `dev` it reached 77% in the first step. Worth watching: it may simply be a cold cache over a far larger key space, or the hot set may be too diffuse at this catalog size even with the skew.

## Open

Expand All @@ -94,6 +88,7 @@ Known gaps, each with why it is still open rather than closed.

## Open decisions

- **Where the tier multipliers live.** They are now normative in `SPEC.md` POST /cart/:id/quote (*Resolved unit price*), which closes the ambiguity. The alternative is `rate` rows discriminated by `kind: 'tier'`: the figures would then be dataset-pinned and checksummed, derivable by any implementation from data it already loads, and the duplicated constant in `resources/lib/pricing.js` and `packages/seed/src/vocabulary.ts` — which nothing currently checks for drift — would go away. Costs one dataset regeneration, which the pricing phase balance decision below likely spends anyway.
- **Facet counting**, when listing pages return. Three candidates identified, none chosen; see [`future-work.md`](future-work.md).
- **Promotion eligibility lookup** — indexed array probes versus one denormalized key. A P1 measurement, not a guess. See [`data-model.md`](data-model.md).
- **Dataset distribution via Git LFS or release assets.** LFS today: the org is on Enterprise with ample headroom and near-zero usage, so the current dataset is comfortable. Release assets are unmetered and deletable, which matters if regeneration becomes routine. Revisit if the dataset grows substantially, regeneration becomes frequent, or clone volume climbs.
Expand Down
2 changes: 1 addition & 1 deletion docs/seed-design.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Dataset generation

Status: **P0 design.** Implements SPEC.md §3.
Status: **P0 design.** Implements [Data model](../SPEC.md#data-model).

## Generate once, commit the artifact

Expand Down
Loading
Loading