Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions benchmarks/locomo/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# RS-LoCoMo-Full-v20 setup
# RS-LoCoMo-Full-v21 setup

This directory contains the unshipped full-system LoCoMo adapter. It does not vendor or
auto-download LoCoMo. Supply the exact pinned `locomo10.json` only after confirming its
Expand All @@ -16,12 +16,12 @@ The safe first command is local and makes no API or model call:
uv run --extra benchmark python -m benchmarks.locomo prepare \
--dataset /absolute/path/locomo10.json \
--tier smoke \
--protocol full-v20 \
--protocol full-v21 \
--output .benchmark-runs/locomo-smoke
```

The harness validates the pinned bytes, renders session documents, and fingerprints the
eight-question smoke plan. `--protocol` is prepare-only; `full-v20` is the one
eight-question smoke plan. `--protocol` is prepare-only; `full-v21` is the one
current-system protocol, and every later stage reads that immutable choice from
`run.json`. Do not run remote stages until reviewing
[`locomo_benchmark_design.md`](../../plan/designs/locomo_benchmark_design.md).
Expand Down Expand Up @@ -64,6 +64,17 @@ call sees the complete bounded candidate snapshot and returns a supplied
candidate id or `new`. There is no insufficient-evidence result or
confidence-routed frontier call.

V21 fingerprints the D106 observation adjudicator: dated events with disjoint
resolved windows never collapse onto or supersede each other (they may only
contradict or stay distinct), a dated event is never `evidence` for an undated
statement (nor the reverse), open-ended windows stay unbounded, and the verdict
prompt shows when each statement was said and what time it is about. The
`adjudicate_observations` component version pins
that generation; the dataset, rendered documents, retrieval surface, answer
and judge prompts, budgets, and scoring are those of v20. V20 and v21 scores
are directional, not a one-variable comparison — the fact layer a v21 store
serves differs from a v20 store's.

Build the image from the revision under test — Compose otherwise serves the
published release image, and the harness refuses to run against an engine whose
stamped revision does not match the prepared run:
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/locomo/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
"""The pinned RS-LoCoMo-Full-v20 benchmark adapter."""
"""The pinned RS-LoCoMo-Full-v21 benchmark adapter."""
2 changes: 1 addition & 1 deletion benchmarks/locomo/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="python -m benchmarks.locomo",
description=(
"RS-LoCoMo-Full-v20: prepare is local; ingest/answer/judge require "
"RS-LoCoMo-Full-v21: prepare is local; ingest/answer/judge require "
"explicit execution acknowledgements"
),
)
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/locomo/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -140,10 +140,10 @@ def load_manifest(tier: str) -> QuestionManifest:
)
if manifest.dataset_commit != DATASET_COMMIT:
raise DatasetValidationError(
"manifest dataset commit is not RS-LoCoMo-Full-v20"
"manifest dataset commit is not RS-LoCoMo-Full-v21"
)
if manifest.dataset_sha256 != DATASET_SHA256:
raise DatasetValidationError("manifest dataset hash is not RS-LoCoMo-Full-v20")
raise DatasetValidationError("manifest dataset hash is not RS-LoCoMo-Full-v21")
actual = item_ids_hash(item_ids=manifest.item_ids)
if actual != manifest.item_ids_sha256:
raise DatasetValidationError(
Expand Down
10 changes: 5 additions & 5 deletions benchmarks/locomo/model.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Typed values for the full-system RS-LoCoMo-Full-v20 protocol."""
"""Typed values for the full-system RS-LoCoMo-Full-v21 protocol."""

from __future__ import annotations

Expand Down Expand Up @@ -28,8 +28,8 @@
Category = Literal[1, 2, 3, 4, 5]
RetainedCategory = Literal[1, 2, 3, 4]
Tier = Literal["smoke", "development", "publication"]
ProtocolKey = Literal["full-v20"]
ProtocolName = Literal["RS-LoCoMo-Full-v20"]
ProtocolKey = Literal["full-v21"]
ProtocolName = Literal["RS-LoCoMo-Full-v21"]
SourceTimezoneBasis = Literal["assumed_utc"]
AnswerAgentModel = Literal["openai/gpt-5.6-luna"]
JudgeModel = Literal["openai/gpt-5.6-luna"]
Expand Down Expand Up @@ -120,7 +120,7 @@ class QuestionManifest(FrozenModel):
class RunConfiguration(FrozenModel):
"""Immutable identity of one prepared benchmark run."""

protocol_name: ProtocolName = "RS-LoCoMo-Full-v20"
protocol_name: ProtocolName = "RS-LoCoMo-Full-v21"
adapter_version: NonEmpty
prepared_at: datetime
repository_revision: NonEmpty
Expand Down Expand Up @@ -393,7 +393,7 @@ class SessionDiagnosticSummary(FrozenModel):
class RunSummary(FrozenModel):
"""Publication-ready local aggregate with no hidden denominator."""

protocol_name: ProtocolName = "RS-LoCoMo-Full-v20"
protocol_name: ProtocolName = "RS-LoCoMo-Full-v21"
protocol_fingerprint: NonEmpty
tier: Tier
questions: int = Field(ge=1)
Expand Down
14 changes: 7 additions & 7 deletions benchmarks/locomo/protocol.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@
from rememberstack.model import Envelope
from rememberstack.model import ToolDescriptor

PROTOCOL_NAME: Final = "RS-LoCoMo-Full-v20"
DEFAULT_PROTOCOL_KEY: Final = "full-v20"
ADAPTER_VERSION: Final = "locomo-full-adapter-2026.09-complete-values-v20"
PROTOCOL_NAME: Final = "RS-LoCoMo-Full-v21"
DEFAULT_PROTOCOL_KEY: Final = "full-v21"
ADAPTER_VERSION: Final = "locomo-full-adapter-2026.09-temporal-gate-v21"
MAX_TOOL_CALLS: Final = 8
MAX_AGENT_CALLS: Final = 9
ANSWER_READER_RETRY_BUDGET: Final = 2
Expand Down Expand Up @@ -74,7 +74,7 @@
"no-types-1:binary-t4-1:document-t0-1"
),
"adjudicate_observations": (
"e3-obs-flush-2026.08a:claim-fanout-1:entity-fanout-1"
"e3-obs-flush-2026.09a:temporal-gate-1:claim-fanout-1:entity-fanout-1"
),
"adjudicate_supersession": "adjudicator-2026.07b:temp0-1",
"embed_claim": "p1-embed-claims-2026.07",
Expand Down Expand Up @@ -211,8 +211,8 @@ class LoCoMoProtocol:
answer_word_cap: int | None = None


_FULL_V20 = LoCoMoProtocol(
key="full-v20",
_FULL_V21 = LoCoMoProtocol(
key="full-v21",
name=PROTOCOL_NAME,
answer_agent_model=ANSWER_AGENT_MODEL,
judge_model=JUDGE_MODEL,
Expand All @@ -234,7 +234,7 @@ class LoCoMoProtocol:
)

PROTOCOL_REGISTRY: Final[Mapping[ProtocolKey, LoCoMoProtocol]] = MappingProxyType(
{_FULL_V20.key: _FULL_V20}
{_FULL_V21.key: _FULL_V21}
)


Expand Down
10 changes: 5 additions & 5 deletions benchmarks/locomo/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,7 @@ def ingest_sample(
_require_current_ingest_bindings(model_bindings=build.model_bindings)
if build.document_binding_generation != EXPECTED_DOCUMENT_BINDING_GENERATION:
raise ExecutionGuardError(
"deployment document binding generation differs from RS-LoCoMo-Full-v20"
"deployment document binding generation differs from RS-LoCoMo-Full-v21"
)
_require_current_query_surface(context=context, client=client)
_require_exact_live_ingests(
Expand Down Expand Up @@ -570,7 +570,7 @@ def answer_sample(
):
raise ExecutionGuardError(
"the deployment did not report the exact completed"
" RS-LoCoMo-Full-v20 pipeline, live graph, and fresh P3 projection"
" RS-LoCoMo-Full-v21 pipeline, live graph, and fresh P3 projection"
)
_require_serving_revision(context=context, readiness=readiness)
prior_readiness = context.state.readiness.get(sample_id)
Expand Down Expand Up @@ -1177,7 +1177,7 @@ def _validate_run(
"""Recompute immutable run identity before any local or remote stage."""
selected_protocol = protocol_for_name(configuration.protocol_name)
if configuration.dataset_sha256 != DATASET_SHA256:
raise BenchmarkRunError("run dataset hash is not RS-LoCoMo-Full-v20")
raise BenchmarkRunError("run dataset hash is not RS-LoCoMo-Full-v21")
if item_ids_hash(item_ids=manifest.item_ids) != manifest.item_ids_sha256:
raise BenchmarkRunError("run manifest item hash changed")
if manifest_bytes_hash(manifest=manifest) != configuration.manifest_sha256:
Expand All @@ -1187,7 +1187,7 @@ def _validate_run(
if manifest.tier != configuration.tier:
raise BenchmarkRunError("run manifest tier changed")
if configuration.dataset_commit != DATASET_COMMIT:
raise BenchmarkRunError("run dataset commit is not RS-LoCoMo-Full-v20")
raise BenchmarkRunError("run dataset commit is not RS-LoCoMo-Full-v21")
if configuration.adapter_version != ADAPTER_VERSION:
raise BenchmarkRunError("run adapter version differs from current code")
if _models_hash(values=documents) != configuration.documents_sha256:
Expand Down Expand Up @@ -1445,7 +1445,7 @@ def _require_current_ingest_bindings(*, model_bindings: dict[str, str]) -> None:
if model_bindings.get(name) != expected.get(name)
)
raise ExecutionGuardError(
"deployment ingest model bindings differ from RS-LoCoMo-Full-v20: "
"deployment ingest model bindings differ from RS-LoCoMo-Full-v21: "
+ ", ".join(mismatches)
)

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/locomo/sharding/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ as source-derived data.
| --- | ---: | --- |
| `LOCOMO_PYTHON` | `.venv/bin/python` | repository virtual-environment Python |
| `LOCOMO_TIER` | `publication` | prepared manifest tier |
| `LOCOMO_PROTOCOL` | `full-v20` | prepare-time protocol key |
| `LOCOMO_PROTOCOL` | `full-v21` | prepare-time protocol key |
| `LOCOMO_MOUNT_ROOT` | `$RUN_DIR/.mounts` | host/container-identical P3 mount root |
| `LOCOMO_MAX_DOCUMENTS` | `100` | per-sample ingest authorization |
| `LOCOMO_MAX_QUESTIONS` | `1540` | run-absolute answer item authorization |
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/locomo/sharding/run_shard.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ run_dir=$2
dataset_path=$3
python_bin=${LOCOMO_PYTHON:-.venv/bin/python}
tier=${LOCOMO_TIER:-publication}
protocol=${LOCOMO_PROTOCOL:-full-v20}
protocol=${LOCOMO_PROTOCOL:-full-v21}
mount_root=${LOCOMO_MOUNT_ROOT:-$run_dir/.mounts}
max_documents=${LOCOMO_MAX_DOCUMENTS:-100}
max_questions=${LOCOMO_MAX_QUESTIONS:-1540}
Expand All @@ -50,7 +50,7 @@ export GOOGLE_APPLICATION_CREDENTIALS=${LOCOMO_GCP_CREDENTIALS_FILE:-/etc/rememb
export GOOGLE_API_CERTIFICATE_CONFIG=${LOCOMO_GCP_CERTIFICATE_CONFIG_FILE:-/etc/rememberstack/locomo-gcs/certificate-config.json}
export GOOGLE_API_USE_CLIENT_CERTIFICATE=true

# RS-LoCoMo-Full-v20's non-secret ingest identity. Override ambient self-host
# RS-LoCoMo-Full-v21's non-secret ingest identity. Override ambient self-host
# defaults so every shard runs the exact Luna/Qwen pipeline the protocol checks.
export REMEMBERSTACK_STRUCTURER_MODEL=openai/gpt-5.6-luna
export REMEMBERSTACK_SKELETON_CHECK_MODEL=openai/gpt-5.6-luna
Expand Down
102 changes: 101 additions & 1 deletion decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -2910,7 +2910,11 @@ may proceed to its first tagged artifact proof after CLA activation.

## D78. LoCoMo measures the ordinary OSS query system, not a claims-only shortcut

> **D105 amendment.** The current protocol is `RS-LoCoMo-Full-v20`. It retains
> **D106 amendment.** The current protocol is `RS-LoCoMo-Full-v21`. It retains
> v20's dataset, rendered documents, models, tools, budgets, prompts, and
> scoring and pins the D106 observation adjudicator generation.

> **Historical D105 amendment.** The D105 protocol was `RS-LoCoMo-Full-v20`. It retains
> v19's surface, dataset, ingestion, models, tools, and budgets while adding the
> D105 complete-direct-values answer instruction.
>
Expand Down Expand Up @@ -4889,3 +4893,99 @@ retry; mention research or adoption in the prompt.
**Amends.** Advances D104's Full-v19 benchmark identity only. It preserves D98
retrieval, D99's objective content-before-`Unknown` guard, and D104's bounded
counterfactual instruction.

## D106. Dated events never collapse across dates; evidence requires temporal compatibility

**Decision (2026-09-03).** The observation adjudicator (D43) gains a
deterministic temporal-compatibility rung that runs before any model call,
reading the D41 valid-time each claim already carries:

1. Two statements that both describe **datable events**
(`claim_valid_kind = 'event_time'`) whose resolved windows are **disjoint**
are different occurrences unless the date-aware model finds they name the
*same* occurrence with disputed dates. Such a pair may therefore only
`contradict` (both stand, grouped) or stay `new`; an `evidence` or
`supersede` verdict is coerced to `new` and recorded. This holds even when
the wording is byte-identical — "won a tournament last week" said in
January and again in October is two wins — and identical text is then
kept apart without any model call.
2. A dated event paired with an **undated** statement (a state, a period
figure, a summary such as "has been winning a few tournaments", or
testimony the extractor could not anchor) may still be judged for
`supersede`/`contradict` — a dated resignation can end an undated "is
CEO" state — but an `evidence` verdict for such a pair is coerced to `new`
and recorded, and identical text is kept apart without a model call. A
specific dated event is never a re-assertion of a vaguer statement, and a
summary never re-asserts a specific event.
3. Two undated statements are adjudicated exactly as before; two dated
events with overlapping windows — including a year-level and a day-level
window for one occurrence — go to the ladder as before, and a re-mention
of the same event still collapses.
4. A D41 open interval (a resolved start with no end) is unbounded, never a
point: it overlaps every later window.

The verdict prompt now shows, for each statement, when the source *said it*
and what world-time it *is about* (the resolved D41 window of any kind), and
states the rule, so the model judges with both timelines in front of it. An
observation's windows are the span of its supporting current-testimony
claims, derived at block time and kept current within a batch as the row
absorbs evidence; nothing is stored. Every adjudication record carries the
coercions that preceded it. The adjudicator generation, the
`adjudicate_observations` flush component version, and the LoCoMo protocol
(Full-v20 → Full-v21) roll; nothing else about D43 — the entity block, the
novelty gate, the no-cap rule, the fail-safe-to-coexist contract — changes.

**Context.** The v0.11.0 LoCoMo `conv-42` run (protocol Full-v18,
`plan/analysis/locomo_conv42_recurring_event_adjudication.md`) extracted all
seven of Nate's tournament wins as clean, dated claims and resolved one Nate
entity, yet the fact layer kept only four distinct win facts. The
`observation_adjudications` transcript shows why: the small model saw two
bare strings, judged "won a really big video game tournament last week"
(October) to be `evidence` for "won his first video game tournament last
week" (January) because both say "last week", and folded the international
and Valorant wins into "has been winning a few gaming tournaments". Ten
participation lineages collapsed into the header boilerplate "Nate is a
participant.", swallowing two real tournament entries. The claims carried
resolved event dates the whole time; the adjudicator never read them. The
reader answered the counting question "At least five" against a gold of
seven with all seven wins in its testimony context — the fact envelope's
undercount is the plausible anchor.

**Consequences.** Recurring same-shaped events survive as distinct facts, so
entity-anchored counting and enumeration have a complete fact layer to read,
while a genuine same-occurrence date dispute can still surface as a
contradiction. Verdict spend is unchanged in shape (similar pairs are still
judged; only identical-text decisions become model-free). The fail-safe
direction is preserved and slightly widened: an undated re-mention of a
dated event now stays a separate observation rather than collapsing — a
duplicate, never a loss. Observation `valid_from` continues to be the
claim's `asserted_at`; using the resolved window as the observation's own
validity is a separate change this decision does not make.

Rollout is stop-drain-rebuild. The `adjudicate_observations` flush handler
reports the claimed unit's *own* component generation to its barrier, so
entity units enqueued before the roll complete under the generation their
barrier counts and drain cleanly; readiness then reports the stage at the old
generation until the deployment is rebuilt under the new one (D7/D12).
Existing stores re-adjudicate observations under the new generation on that
rebuild.

**Rejected.** Fixing only the verdict prompt (a model may still merge across
dates; the coercions must be deterministic); skipping disjoint-date pairs
without any verdict (would silently double-count one occurrence whose date
two sources dispute, and suppress the contradiction group); treating every
mixed-dating pair as non-interacting (would stop a dated event from
superseding a state it ends); dropping vague summaries at extraction
(source-faithful testimony is kept — D32 — the defect was absorption, not
extraction); a typed period/value column (D43 §4 rejects the typed schema;
the event window is already carried by the claim).

**Design.** `plan/designs/observations_design.md` §3 (temporal compatibility
rung) and `plan/designs/locomo_benchmark_design.md` Full-v21 protocol.

**Analysis.** `plan/analysis/locomo_conv42_recurring_event_adjudication.md`.

**Amends.** D43's adjudication cascade (adds the deterministic rung its
design named but never implemented) and D105's Full-v20 benchmark identity.
Preserves D41 (claims stay the one home of asserted validity), D98
retrieval, D100/D102 identity, and D104/D105 answer-prompt contracts.
Loading
Loading