A quality gate for generated training data

doc/training.md

@@ -66,6 +66,58 @@ python -m sciencebeam_parser.training.cli.generate_data \
 
 Additionally the `--gzip` argument can be passed in, resulting in gzip (`.gz`) compressed output files.
 
+#### The quality record
+
+Every run writes a `quality.jsonl` per model it generated for, one JSON line per
+source document, whether the document succeeded or not. It holds the count at each
+stage where the cardinality of the labels can change, so that a corpus can be
+compared against the JATS it was aligned from without counting labels in the
+generated data afterwards.
+
+The record is per model because generation is run per model: a corpus commonly
+holds one model's data at one document set and another model's at a different one,
+and a record covering the whole corpus would describe the last run rather than the
+data beside it. With `--use-directory-structure` each file sits beside that model's
+`corpus` directory, otherwise it is `<model>.quality.jsonl` in the output path:
+
+```text
+reference-segmenter/quality.jsonl
+citation/quality.jsonl
+```
+
+```json
+{
+  "document_id": "PPR459453",
+  "source_filename": "PPR459453.pdf",
+  "status": "ok",
+  "model": "citation",
+  "jats": {"status": "ok", "reference_count": 45, "aligned_reference_count": 2},
+  "written": true,
+  "entity_element_count": 2,
+  "label_counts": {"<title>": {"jats": 44, "marked": 2}}
+}
+```
+
+- `jats.status` is `ok`, `missing` (no JATS was matched), `unparsable` or
+  `unreadable`. A `reference_count` of 0 with status `ok` is a JATS that declares
+  no references — there was never anything to align.
+- `aligned_reference_count` is how many of those references the aligner placed, so
+  the difference from `reference_count` is alignment's.
+- `entity_element_count` is what the model wrote per entity: `bibl` for
+  `reference-segmenter` and `citation`, and absent for a model whose labels mark
+  regions rather than repeated entities. `written: false` is a model that found no
+  entities and so wrote no file at all.
+- `label_counts` is per citation label, over references rather than occurrences:
+  `jats` counts references whose JATS carries a sub-field for that label, `marked`
+  counts references the training data marks it in. The two differ legitimately —
+  a printed reference does not carry everything its JATS does, so a low rate for
+  an identifier or a URL is usually the page rather than the pipeline.
+
+The record is written by the parent process as each document finishes, so a run
+that is interrupted keeps the records it had, and a document that timed out or
+failed is present with a `status` of `timeout` or `error` in every model's file
+rather than missing.
+
 ### Annotating `tei` training data for the sequence models
 
 After the `tei` training data has been generated, it should get reviewed and manually annotated.
@@ -94,6 +146,87 @@ which `delft` models read as a text feature and `wapiti` templates do not
 reference. Training data generated with the flag cannot be mixed with GROBID's
 `segmentation` corpus, since it is one column wider.
 
+#### The assembly quality record
+
+This step is the only place the last count exists: the TEI holds elements, and
+what the training data ends up with is the entities those elements parse back to.
+It writes `<delft-output-path>.quality.jsonl` (or `--quality-output-path`), one row
+per document, and logs a summary per corpus.
+
+Pass `--quality-record-path` to join what generation recorded, so that a loss can be
+attributed to a stage rather than only observed:
+
+```bash
+python -m sciencebeam_parser.training.cli.generate_delft_data \
+    --model-name="reference_segmenter" \
+    --tei-source-path="data/generated-training-data/train/*/reference-segmenter/corpus/tei/*.tei.xml" \
+    --quality-record-path="data/generated-training-data/train/*/reference-segmenter/quality.jsonl" \
+    --delft-output-path="./data/generated-training-data/delft/reference-segmenter/corpus/reference-segmenter.data"
+```
+
+```json
+{
+  "document_id": "PPR459453", "model": "reference-segmenter", "corpus": "scielo_preprints-jats",
+  "sequence_count": 1, "entity_start_count": 2,
+  "generated": {"jats": {"reference_count": 45}, "entity_element_count": 2}
+}
+```
+
+- `entity_start_count` against the generated `entity_element_count` is the parse: fewer
+  entities than elements is a boundary lost between siblings, and the summary names
+  the documents it happened to.
+- `sequence_count` is the training sequences the document contributes. For `citation`
+  every element is its own sequence, so it has no entity count and carries
+  `label_start_counts` instead — per label, the sequences marking it, comparable with
+  what generation recorded as marked.
+- A document generation recorded that produced no training data at all is reported
+  by id: nothing that iterates the TEI can see it.
+
+Without `--quality-record-path` the counts are still recorded, with nothing to
+compare them against — which is enough to re-check a committed corpus offline.
+
+#### Filtering on quality
+
+`--quality-filter` leaves out the documents that fail the thresholds in
+[`training_quality.yml`](../sciencebeam_parser/resources/training_quality.yml),
+rather than only recording their counts. Each exclusion is reported with the stage
+that failed and the numbers behind it, and the summary says what was kept per
+corpus:
+
+```text
+excluding 5-264_v2: excluded (jats-not-readable) [jats_status=unparsable]
+excluding PPR459453: excluded (elements-short-of-jats)
+  [element_ratio=0.044, entity_element_count=2, jats_reference_count=45]
+ore / reference-segmenter: kept 37 of 38 documents (3% excluded);
+  jats-not-readable: ['5-264_v2']
+```
+
+The same reasons are written to each row of the record, so a corpus that shrinks
+can always be accounted for document by document.
+
+Assembly **refuses** rather than proceeds when a corpus loses more than
+`corpus.max_excluded_ratio` of its documents: dropping most of a corpus is a
+finding about the pipeline, not a routine filter outcome. `--max-excluded-ratio`
+overrides it for a run, which is a decision to record rather than a way around
+the refusal.
+
+Thresholds are per model, and a model with no entry in the config fails rather
+than being assumed sound. Models whose labels mark regions instead of repeated
+entities carry `cardinality: none` and a reason. Two things are deliberately
+recorded rather than failed:
+
+- **more elements than the JATS has references.** The reference segmenter writes a
+  block per contiguous run of a reference's lines, so a reference split across a
+  column becomes two elements — where the citation model's count of the same
+  references matches the JATS exactly. The document is otherwise sound.
+- **citation label rates.** No floor is set, because the level of a label's rate
+  says as much about the publisher as the pipeline: ORE prints no DOI at all where
+  its JATS carries one for 1205 of 1679 references, so a floor on the identifier
+  would reject nearly every ORE document. A rate that moves is the finding.
+
+Filtering is off unless asked for, since this command is also run over corpora
+that carry no record at all, such as GROBID's own.
+
 #### Example command for `segmentation` model
 
 ```bash

sciencebeam_parser/resources/training_quality.yml

@@ -0,0 +1,86 @@
+# What the quality gate requires of a generated document before its training
+# data is used, per model.
+#
+# The counts come from the quality record: the references the JATS declares, the
+# elements generation wrote, and the entities those elements parse back to.  Each
+# threshold is a floor on one stage against the one before it, so a document that
+# fails names the stage that lost its references rather than only that it is wrong.
+#
+#   min_jats_reference_count  a document whose JATS declares fewer references than
+#                             this has no reference of record; no comparison
+#                             between stages can flag it, because every stage
+#                             agrees on zero
+#   min_element_ratio         elements written, over the references the JATS
+#                             declares.  An exact match is too strict: what a
+#                             PDF's reference list contains and what the JATS
+#                             carries legitimately differ
+#   min_entity_ratio          entities the training data ends up with, over the
+#                             elements the TEI holds.  This is the stage where two
+#                             sibling elements sharing a label used to come back as
+#                             one entity, and the floor is what keeps that from
+#                             returning unnoticed
+#
+# Over-counting is recorded and not failed.  The reference segmenter writes a
+# block per contiguous run of a reference's lines, so a reference split across a
+# column or a page becomes two elements -- 7 of 50 scielo_preprints-jats documents,
+# where the citation model's count of the same references matches the JATS exactly.
+# Those documents are otherwise sound, and the split is a defect of alignment
+# rather than of the data's cardinality.
+#
+# A model with no cardinality to check carries `cardinality: none` and a reason,
+# rather than being left out: a missing entry is a decision not taken, and the
+# gate says so instead of passing silently.
+
+corpus:
+  # Beyond this share of a corpus excluded, assembly refuses rather than
+  # proceeds.  Dropping most of a corpus is a finding about the pipeline, not a
+  # routine filter outcome, and the counterpart of failing on inconsistent
+  # feature lengths on the training side.  Raise it deliberately, with the
+  # reason recorded, rather than working around the refusal.
+  max_excluded_ratio: 0.2
+
+models:
+  reference-segmenter:
+    min_jats_reference_count: 1
+    min_element_ratio: 0.9
+    min_entity_ratio: 0.8
+
+  citation:
+    min_jats_reference_count: 1
+    min_element_ratio: 0.9
+    # Every element is its own training sequence, so there is no entity ratio to
+    # take; what can change is which labels are marked, and that is recorded per
+    # label rather than gated.  No floor is set because the level of a label's
+    # rate says as much about the publisher as the pipeline: title, journal,
+    # date, volume, pages and author sit at 0.95 and above on both measured
+    # corpora, while ORE prints no DOI at all where its JATS carries one for
+    # 1205 of 1679 references, so a floor on the identifier would reject nearly
+    # every one of its documents.  A rate that moves is the finding.
+    label_floors: {}
+
+  segmentation:
+    cardinality: none
+    reason: labels mark regions of a document, which occur once rather than repeatedly
+  header:
+    cardinality: none
+    reason: labels mark fields of one header, so presence is what matters
+  affiliation-address:
+    cardinality: none
+    reason: >
+      one entity per affiliation, but the JATS count is of affiliations rather than
+      references and has no measured threshold yet
+  name-header:
+    cardinality: none
+    reason: one entity per author name, with no measured threshold yet
+  name-citation:
+    cardinality: none
+    reason: one entity per author name, with no measured threshold yet
+  fulltext:
+    cardinality: none
+    reason: labels mark body regions, which occur once each
+  figure:
+    cardinality: none
+    reason: one entity per figure, with no measured threshold yet
+  table:
+    cardinality: none
+    reason: one entity per table, with no measured threshold yet