From d3b5488777787ec320e1d02815b8996d5826b7ef Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sun, 9 Aug 2026 13:02:01 +0100 Subject: [PATCH] meta docs: catch up with the pack migration and the model changes since data-model.md still documented the retired file-per-record layout (works///work.json and friends) as current storage. It now describes the range-packed layout - slug is identity, the pack file is only storage, file names are range bounds, metafmt self-heals placement - and catches up with the model changes that landed since: work genres and role-qualified credits, added_at, the bare-or-region-scoped recording isbn[] and regional publishers[], person kind, the libex-import source type, and artifact schema_version 4 (work_genres) with its serve-side gating. overview.md: the data/ tree line, pkg/model's description and the new pkg/pack, metacheck/metafmt/metabuild rows (pack invariants, self-heal, added_at from the data instead of the retired --added git walk), and the Dockerfile line - the image ships no baked data since the poll-only boot landed. contributing-data.md: the check.yml description's id/shard-agreement wording. --- docs-developers/meta/contributing-data.md | 6 +- docs-developers/meta/data-model.md | 101 ++++++++++++++-------- docs-developers/meta/overview.md | 15 ++-- 3 files changed, 76 insertions(+), 46 deletions(-) diff --git a/docs-developers/meta/contributing-data.md b/docs-developers/meta/contributing-data.md index 4db3e9e..298c308 100644 --- a/docs-developers/meta/contributing-data.md +++ b/docs-developers/meta/contributing-data.md @@ -82,9 +82,9 @@ the workspace-wide CI rules. Every pull request touching `data/**` is checked twice: - **`check.yml` (mechanical, blocking).** Runs `go build`/`vet`/`test`, - `metacheck` (schema, id/shard agreement, referential integrity, uniqueness, - chapter/series rules), and `metafmt --check` (canonical JSON). A red pull - request never merges. It uses the plain `pull_request` trigger, so fork pull + `metacheck` (schema, pack placement and caps, referential integrity, + uniqueness, chapter/series rules), and `metafmt --check` (canonical JSON + + pack invariants). A red pull request never merges. It uses the plain `pull_request` trigger, so fork pull requests run with a read-only token and no secrets. - **`ai-verify.yml` (advisory, never blocking).** An AI judgement layer on top of the mechanical check: it posts a `PASS` / `FLAG` comment and label but **never diff --git a/docs-developers/meta/data-model.md b/docs-developers/meta/data-model.md index 2fef18e..d2b8932 100644 --- a/docs-developers/meta/data-model.md +++ b/docs-developers/meta/data-model.md @@ -1,16 +1,26 @@ --- title: Meta data model -description: "The audiosilo-meta entities and their on-disk layout: path-is-identity slugs and sharding, works/recordings/people/series, the characters and recaps sidecars, the position model, provenance, and the JSON Schema contract." +description: "The audiosilo-meta entities and their on-disk layout: slug identity and range-packed storage, works/recordings/people/series, the characters and recaps sidecars, the position model, provenance, and the JSON Schema contract." --- -## Path is identity +## Slug is identity; the file is only storage Every entity is addressed by a **slug** matching `^[a-z0-9]+(-[a-z0-9]+)*$` -(`common.schema.json` `$defs/slug`, max 100 chars), and its file lives in a -**shard directory** named for the first two characters of the slug. Sharding -keeps any one directory small as the catalogue grows. The slug is the identity - -there is no numeric id anywhere in the data, and the file path encodes exactly -where a record lives. +(`common.schema.json` `$defs/slug`, max 100 chars). The slug is the identity - +there is no numeric id anywhere in the data. + +Storage is **range-packed** (the repo's `PACK-SPEC.md` is the full +specification): each of the four families - `data/works/`, `data/works-community/`, +`data/people/`, `data/series/` - is a set of **pack files**, each holding many +records in an `entries` map keyed by slug. A pack file's name is only the **range +bound** its slugs start at, never the identity of anything inside it: +`data/series/wheel-of-time.json` holds every series whose slug falls in its +range, which is mostly not the Wheel of Time. Nobody computes placement by hand - +`metafmt --write` relocates a misplaced entry, performs due splits, and +re-renders canonically, so an approximately-right edit is corrected +mechanically. (The pre-migration layout was one file per record, +`works///work.json`; those paths survive only as a reference +syntax the issue forms still accept.) The JSON Schemas in `schema/*.schema.json` (JSON Schema draft 2020-12, every object `additionalProperties: false`) are the **authoritative, public contract**. @@ -20,9 +30,12 @@ schemas. ## The factual core (CC0) -### work - `data/works///work.json` +### work - an entry in the `data/works/` family -The abstract book, independent of any particular narration. +The abstract book, independent of any particular narration - stored as a +**composite**: the work's own fields plus its recordings nested as +`"recordings": {"": {...}}`, so one book is one entry and a recording +edit is a read-modify-write of its work. | Field | Required | Notes | |---|---|---| @@ -33,16 +46,21 @@ The abstract book, independent of any particular narration. | `language` | yes | BCP-47-ish (`^[a-z]{2,3}(-[a-z0-9]{2,8})*$`) | | `first_published` | no | `YYYY` or `YYYY-MM-DD` | | `description` | no | community-written, never a publisher blurb | +| `genres` | no | values from the project's controlled vocabulary (`$defs/genre`, a flat retailer-neutral list), sorted ascending | +| `credits` | no | role-qualified contributors as `{person, role}` pairs; `role` from the `$defs/credit_role` enum (adaptation, afterword, contributor, editor, foreword, illustrator, introduction, preface, translator). Additive and parallel to `authors`; emitted only when a source stated the role | | `xref` | no | `wikidata` (`Q\d+`), `openlibrary` (`OL\d+W`), `goodreads`, print `isbn[]` | +| `added_at` | no | `YYYY-MM-DD`, or a full RFC 3339 timestamp for migration-backfilled records | +| `recordings` | yes | the nested map of this work's recordings (below) | | `license` | yes | `CC0-1.0` | | `sources` | yes | provenance (below) | -### recording - `data/works///recordings/.json` +### recording - a member of its work entry's `recordings` map A specific narration/production of a work. **One work, many recordings** - the canonical example is *Harry Potter and the Philosopher's Stone*, one work with a -Stephen Fry recording and a Jim Dale recording, each carrying its own ASINs. The -shard is the **parent work's** slug shard. +Stephen Fry recording and a Jim Dale recording, each carrying its own ASINs. A +production released in several marketplaces stays **one recording**: the region +rides on the identifiers and the imprint, never on a second record. | Field | Required | Notes | |---|---|---| @@ -53,22 +71,27 @@ shard is the **parent work's** slug shard. | `language` | yes | | | `runtime_min` | no | integer minutes, > 0 | | `release_date` | no | `YYYY`, `YYYY-MM`, or `YYYY-MM-DD` | -| `publisher` | no | | | `asin` | no | array of `{region, asin}`; `region` is one of 11 storefronts (`us`, `uk`, `ca`, `au`, `de`, `fr`, `es`, `it`, `jp`, `in`, `br`); `asin` is `[A-Z0-9]{10}` | -| `isbn` | no | array of bare 10/13-digit ISBNs | +| `isbn` | no | array whose entries are a bare 10/13-digit ISBN string (region unstated - the scale form every bulk writer emits) or the object `{isbn, region}` when the marketplace is known; uniqueness keys on the value, so both spellings of one identifier collide | +| `publisher` | no | the publisher of record | +| `publishers` | no | array of `{region, publisher}`: other regions' imprints. May never restate the top-level `publisher` or name one region twice | | `cover_url` | no | must be an `https://` URL | | `chapters` | no | array of `{title, start_ms, length_ms}` | +| `added_at` | no | as on work | | `license` | yes | `CC0-1.0` | | `sources` | yes | | -### person - `data/people//.json` +### person - an entry in the `data/people/` family -One human, shared across roles: authors on works and narrators on recordings are +One person, shared across roles: authors on works and narrators on recordings are the same entity type, and a person can be both. Fields: `id`, `name`, optional -`sort_name`, optional `description`, optional `xref` (`wikidata`, `openlibrary` -`OL\d+A`, `audible` ASIN), `license` (`CC0-1.0`), `sources`. +`sort_name`, optional `kind` (`person` / `group` / `publisher` - marks records +that are not an individual, such as a full cast or a corporate credit of record; +**absence means person or unclassified**, never a guess), optional `description`, +optional `xref` (`wikidata`, `openlibrary` `OL\d+A`, `audible` ASIN), `license` +(`CC0-1.0`), `sources`. -### series - `data/series//.json` +### series - an entry in the `data/series/` family A named, ordered set of works. Each entry is `{work, position}`, where **`position` is a string** so it can express decimals and omnibus ranges: @@ -79,16 +102,19 @@ optional `xref` (`wikidata`, `goodreads`), `license` (`CC0-1.0`), `sources`. ## The expressive layer (CC BY-SA) -Two **per-work sidecars** carry the community-authored, spoiler-tagged content. -They are structurally separated from the core: their `license` field accepts only -`CC-BY-SA-3.0` (`$defs/license_content`). Authoring them is documented in the -repo's `AUTHORING.md` (see [contributing data](contributing-data.md)). +Two **per-work sidecars** carry the community-authored, spoiler-tagged content: +the `characters` and `recaps` members of a work's entry in the separate +`data/works-community/` family, keyed by the **work's** slug. They are +structurally separated from the core: their `license` field accepts only +`CC-BY-SA-3.0` (`$defs/license_content`), and the family boundary makes the +licensing split visible in the directory tree. Authoring them is documented in +the repo's `AUTHORING.md` (see [contributing data](contributing-data.md)). -### characters - `data/works///characters.json` +### characters - the `characters` member of a works-community entry An array of character entries under a `work` slug. Each character has: -- `id` - unique **within the file**, not globally (two works may each have a +- `id` - unique **within the member**, not globally (two works may each have a `bilbo-baggins`); - `name`, optional `aliases[]`, optional `role` (`protagonist` / `antagonist` / `supporting` / `minor`); @@ -97,24 +123,24 @@ An array of character entries under a `work` slug. Each character has: - optional `description` - own-words, length-capped at 1500 chars (a card without one simply has nothing to reveal); - optional `xref` (`wikidata`, `goodreads`) - a shared `wikidata` QID links a - recurring character across a series' per-work files. + recurring character across a series' per-work entries. Recurring characters are **re-described per book**, so what a reader sees stays bounded by which book they are currently in. -### recaps - `data/works///recaps.json` +### recaps - the `recaps` member of the same works-community entry Position-keyed "story so far" summaries under a `work` slug, plus two optional whole-book summaries. Each recap entry has: - `through` - a [position](#the-position-model): the recap is safe to show once - the listener has finished that chapter. **No two recaps in a file share a + the listener has finished that chapter. **No two recaps in a member share a `through` chapter.** - optional `scope` (`book` / `series`) - a `chapter: 0` + `series` entry is the "previously, in earlier books" recap; - `text` - own-words, length-capped at 3000 chars. -The file also carries two optional whole-book fields for a reader who has +The member also carries two optional whole-book fields for a reader who has finished the book: `in_short` (the whole arc in one paragraph, ending included, cap 1500) and `ending` (how the book closes, stated plainly, cap 2000 - deliberately tighter than a chaptered recap entry, a crisp sequel-handoff). @@ -139,7 +165,8 @@ scope. The object shape is deliberately extensible - a later `paragraph` or Every record carries a `sources[]` array (`$defs/sources`, at least one entry). Each source is `{type, ref?, imported_at?}`, where `type` is one of a fixed enum (`user`, `openaudible-import`, `libation-import`, `audiosilo-books-import`, -`audible-lookup`, `openlibrary`, `wikidata`, `inventaire`, `community`) and +`libex-import`, `audible-lookup`, `openlibrary`, `wikidata`, `inventaire`, +`community`) and `imported_at` is `YYYY-MM-DD`. Because every fact records where it came from, a whole source can be audited or retracted. @@ -155,12 +182,14 @@ expressive layer was added in later versions: | 1 | the factual core (works, recordings, people, series, FTS5 search index) | | 2 | the `characters`, `character_aliases`, and `recaps` tables | | 3 | the per-work `recap_summaries` table (the `in_short` / `ending` fields) | - -`metaserve` returns characters and recaps inline on `GET /works/{id}` -(`characters` / `recaps` / `recap_summary`, all `omitempty`). The serve queries -**degrade gracefully** when a newer binary briefly serves an older release: the -characters/recaps queries no-op below `schema_version` 2 and the recap summary -below 3, so a missing table reads as "no data", never a 500. The same versioning +| 4 | the `work_genres` set table | + +`metaserve` returns characters, recaps and genres inline on `GET /works/{id}` +(`characters` / `recaps` / `recap_summary` / `genres`, all `omitempty`). The +serve queries **degrade gracefully** when a newer binary briefly serves an older +release: the characters/recaps queries no-op below `schema_version` 2, the recap +summary below 3, and genres below 4, so a missing table reads as "no data", +never a 500. The same versioning drives the [coverage endpoints](api.md#coverage-endpoints), which omit a dimension's count rather than report it as a misleading zero when the artifact predates its table. diff --git a/docs-developers/meta/overview.md b/docs-developers/meta/overview.md index e4a201c..0510dec 100644 --- a/docs-developers/meta/overview.md +++ b/docs-developers/meta/overview.md @@ -56,20 +56,21 @@ public `pkg/*` (consumed by the sibling `audiosilo-sidecars` module as ordinary dependencies) and the private `internal/*`. ``` -data/ the database: works/, people/, series/ (sharded JSON) + per-work sidecars +data/ the database, range-packed (PACK-SPEC.md): works/ (composites), works-community/ (the CC BY-SA sidecars), people/, series/ schema/ JSON Schemas (one per entity) - the public contract, embedded via schema.go cmd/ thin CLIs: metacheck, metafmt, metabuild, metaserve, metascan, metaimport, metaissue, metaextract (flag wiring only) -pkg/model PUBLIC entity structs, slug/shard rules, location parsing +pkg/model PUBLIC entity structs, slug rules, reserved slugs, pack addressing +pkg/pack PUBLIC pack-file storage: bounds, splits, and the read-through Store every writer uses pkg/canonical PUBLIC canonical JSON (sorted keys, 2-space indent, trailing LF) -pkg/check PUBLIC schema validation + integrity/uniqueness/chapter/series rules +pkg/check PUBLIC schema validation + pack-storage invariants + integrity/uniqueness/chapter/series rules pkg/extract PUBLIC epub split + the word-shingle near-verbatim check pkg/scan PUBLIC local folder scanner (tags + path/filename heuristics + ffprobe) internal/importer OpenAudible / Libation export -> canonical records (ASIN dedup) internal/issueform issue-form body -> canonical records + an ok/duplicate/needs-human/invalid verdict internal/build the deterministic SQLite builder (FTS5, ASIN/ISBN indexes, added_at) internal/serve the read-only HTTP API + ABS provider + GitHub-release poller/hot-swap -Dockerfile image: the site build + the metaserve binary - no data (see below) +Dockerfile image: the site build + the metaserve binary - no baked data; the catalogue is fetched from the newest data release at boot (see below) .github/ issue forms + CI workflows (check, release, image, intake, ai-verify) ``` @@ -84,9 +85,9 @@ run ./cmd/`. | Command | What it does | |---|---| -| `metacheck` | Validates the whole `data/` tree - schema, id/shard agreement, referential integrity, uniqueness, chapter ordering, series positions. Prints one line per problem and exits 1 if any are found. | -| `metafmt` | Enforces canonical JSON for `data/**/*.json` (sorted keys, 2-space indent, single trailing LF). `--check` lists non-canonical files and exits 1; `--write` rewrites them. | -| `metabuild` | Compiles `data/` into the SQLite artifact (`-o meta.sqlite`). Runs the full validation first and refuses to build invalid data. Deterministic: identical data produces an identical artifact. | +| `metacheck` | Validates the whole `data/` tree - schema, pack placement and caps, referential integrity, uniqueness, chapter ordering, series positions. Prints one line per problem and exits 1 if any are found. | +| `metafmt` | Enforces canonical JSON for `data/**/*.json` (sorted keys, 2-space indent, single trailing LF) and the pack-storage invariants: `--write` also relocates misplaced entries, performs due pack splits, and rebinds, so placement self-heals and nobody computes it by hand. `--check` reports and exits 1. | +| `metabuild` | Compiles `data/` into the SQLite artifact (`-o meta.sqlite`). Runs the full validation first and refuses to build invalid data. Deterministic: identical data produces an identical artifact; each record is dated by its own `added_at`, falling back to its newest `sources[].imported_at`. | | `metaserve` | Serves the compiled artifact read-only over HTTP (and optionally the static site at `/`), hot-swapping newer GitHub releases. See [the HTTP API](api.md). | | `metascan` | Scans a local audiobook folder into an import JSON - see [contributing data](contributing-data.md#scanning-local-files-metascan). | | `metaimport` | Ingests an OpenAudible/Libation library export into `data/` - see [contributing data](contributing-data.md#bulk-importers-metaimport). |