A quality gate for generated training data - #701
Merged
Conversation
Nothing checked that a generated document's labels agreed with the JATS they were aligned from. The reference segmenter trained on a corpus where roughly half the documents presented their whole reference list as one reference, and the only sign of it was a coverage ratio in a log line; finding it meant counting labels in the artifact afterwards. Every run now writes a quality.jsonl per model it generated for, one row per source document, holding the count at each stage where the cardinality can change: the references the JATS declares, how many of them the aligner placed, and what the model wrote per entity. The reference of record is the set JatsFieldExtractor emits - back/ref-list/ref with non-empty text - since a peer-review sub-article's own ref-list is never offered to the aligner, and counting every ref-list instead reports a shortfall the pipeline never had: 1708 references for ore against the 1679 it is given. Per model rather than per corpus because generation is run per model. A corpus commonly holds one model's data at one document set and another model's at a different one - the committed corpora carry the reference segmenter and citation at 38 and 50 documents against 5 and 10 for segmentation - so one file per corpus would describe the last run rather than the data beside it. The citation model takes a per-label count rather than a third entity count, because its parser's root element path is bibl, so every element is its own training sequence and the start count equals the element count by construction. Presence per reference on both sides, which holds whichever convention a label uses: <author> covers a whole author list where <pages> is written once per page number. The two sides differ legitimately - ore marks no identifier at all because 2% of its printed references carry a DOI-like string while its JATS carries one for 1205 of 1679 - so this is a rate to compare across regenerations rather than something to expect. The record is written by the parent process as each document finishes, so a document that timed out or died is present in every model's file with its status rather than missing, a document that produced no file at all is visible where nothing iterating generated output would show it, and an interrupted run keeps what it had. A JATS that will not parse is its own status, distinct from one that declares no references. Measuring only: no threshold, no verdict, and nothing is filtered. Over the committed corpora the counts come out at 1679 references against 1678 elements for ore and 1674 against 1648 for scielo_preprints-jats, naming both documents whose JATS declares no references, the one whose JATS will not parse, PPR459453 at 45 references to 2 elements, and the seven documents holding more elements than their JATS has references.
Generation records the references the JATS declares and the elements it wrote, and stops there: the count that decides what a model trains on is the entities those elements parse back to, and only the delft conversion sees them. Without it a loss is visible but not attributable - scielo_preprints-jats looks 26 elements short of its JATS where alignment loses 47, element writing gains 21 to split references, and the parse loses 5. generate_delft_data now takes that count, joins the record generation wrote via --quality-record-path, writes one row per document to <delft-output-path>.quality.jsonl, and logs a summary per corpus naming the documents that lost entities. It reads the generation record rather than extending it, so generated output stays reproducible from generation alone and can be assembled more than once. The citation model has no entity count to take, since its parser's root element path is bibl and every element is already its own training sequence. It records label starts per sequence instead, which is comparable with what generation recorded as marked, and is where spec 010's defect reached that model. Over both corpora the two sides agree exactly, label by label. Three things this could not have been trusted to do without running it: The two CLIs disagree on model names - generation writes reference-segmenter where this one takes --model-name=reference_segmenter - so every lookup keyed on one spelling returned no counts at all while appearing to work. Names are canonicalised, and the record always carries the hyphenated form so the two sides join. Documents that produced no training data were inferred from the files the glob matched, which reported every document outside a narrowed run: 35 of 40 on a three-file run. It reads the record's own `written` instead. Nothing was printed. The import chain installs a root log handler and raises the root level to ERROR, so basicConfig is a no-op and the summary was visible only under --debug; this CLI now sets its own loggers. Counting only: no threshold, no verdict, and nothing is filtered.
The record measured; nothing acted on it. A corpus half of whose documents present their reference list as one reference could still reach a training run, which is what happened. generate_delft_data now takes --quality-filter, leaving out the documents that fail the thresholds in resources/training_quality.yml and reporting each with the stage that failed and the numbers behind it, in the log and in the row the training run can read. Assembly refuses rather than proceeds when a corpus loses more than a configured share of its documents: --max-excluded-ratio overrides that for a run, which is a decision to record rather than a way around it. Thresholds are per model, and a model with no entry fails rather than being assumed sound; the models whose labels mark regions rather than repeated entities carry cardinality: none and a reason. Two things are recorded rather than failed, both measured rather than assumed. More elements than the JATS has references is the reference segmenter writing a block per contiguous run of a reference's lines, so a reference split across a column becomes two elements where the citation model's count of the same references matches the JATS exactly; those seven scielo_preprints-jats documents are otherwise sound and the split belongs to alignment. And no citation label carries a floor, because the level of a label's rate says as much about the publisher as the pipeline: ORE prints no DOI at all where its JATS carries one for 1205 of 1679 references, so a floor on the identifier would reject nearly every one of its documents. A rate that moves is the finding. Filtering is off unless asked for, since this command is also run over corpora that carry no record at all, and what cannot be checked is reported as unchecked rather than assumed good. Over the committed corpora it keeps 37 of 38 ore documents and 49 of 50 scielo_preprints-jats, excluding 5-264_v2 for a JATS that will not parse -- 29 unlabelled citation references, since region and reference boundaries come from the models while labels come from the JATS -- and PPR459453 for a reference region holding 2 elements against 45 references. The reference segmenter's data goes from 88 documents to 86 and citation's from 3335 sequences to 3304. log_on_exception dropped the return value of whatever it wrapped, which is invisible for a test method and silently returns None for a helper in a decorated class.
Three defects in the gate, from reviewing it rather than running it on the two corpora that happen to exercise none of them. A JATS status of `missing` means none was matched, which is what generating without --source-xml-path records for every document. The gate read any status other than ok as a JATS that could not be used, so such a corpus failed document by document and was then refused whole, for a reason that was not true. Only unparsable and unreadable are a defect now: a JATS that was never there leaves nothing to check against, and what cannot be checked is not a failure. The data is written as documents are assembled, so a refused corpus had already been written by the time the corpus-loss check ran, and whatever read that path next would find what looks like a corpus - the silent shrinkage the refusal exists to prevent. The output is now removed when assembly refuses, and the quality record kept so the refusal can be accounted for. A remote path cannot be removed by the writer in use, and says so rather than implying it was. no-training-sequences was collected before the stages that precede it, so a document failing both an earlier stage and that one reported the later one as its primary reason, against what the verdict promises. Stages are built in order. Also asserted: the models the config declares as having no cardinality are the models the counting module declares the same way. Two lists of one fact drift, and the gate would then count a cardinality it never checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/148