Separate empty claims from failed measurements, and count relations once - #1
Merged
FilipKalcic1 merged 1 commit intoSep 19, 2026
Conversation
A claim measured against a table with no rows has not failed. There was nothing to measure, and the claim came out neither supported nor refuted. Reporting that as "unverifiable" reads as a measurement that could not be taken, and it buries the claims that really are still open. A measurement that finds nothing to measure is now marked empty and gets its own verdict. Timeouts, exhausted budgets, SQL errors and claims naming a table that does not exist stay unverifiable. Verdicts carry the reason, so the summary collapses the empty ones into one line grouped by reason, and the write prompt says an empty claim is not a finding and must not be listed one by one. Exit codes do not move. inconsistent_values had the same hole and a worse consequence. An empty column gives zero distinct values and zero canonical forms, so collisions is zero, and decide read that as "rejected". Rejected claims never reach the written files, so the tool was reporting a column as consistent when it had measured nothing at all. Second: the generated README could state a table count taken from the number of files being written. The write prompt asked for a paragraph about the database and left the count to the model, while the run summary counted relations from the catalog. describeKinds moves from cli.ts into verdict.ts, assemble puts its result on Verified as "relations", and the prompt says to use that string as given and never to count relations itself. Relations dropped to fit the budget or the model's input limit are counted on the end of that string, so the README never claims more than was examined. Tested against the fixture: the empty cars table gives a relationship claim and an inconsistent_values suspicion that both come back empty. A live run wrote 13 files and still described the database by its 11 relations. 50 offline tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two fixes you agreed to take. One commit, both tested against the fixture.
"Unverifiable" was hiding two different outcomes
When a claim is measured against a table with no rows, nothing failed. There
was nothing to measure, and the claim came out neither supported nor refuted.
Calling that
unverifiablereads like a measurement the tool could not take.On a schema with many empty or lightly used tables it fills the output with
what look like failures, and buries the claims that really are still open.
A measurement that finds nothing to measure is now marked
emptyand getsits own verdict. Timeouts, exhausted budgets, SQL errors and claims naming a
table that does not exist stay
unverifiable, because those are genuinelyunresolved.
Verdicts carry the reason now, so the summary collapses them into one line
grouped by reason. From a fixture run:
The write prompt says an
emptyclaim is not a finding and must not belisted one by one. Exit codes are untouched.
While separating the two I found
inconsistent_valueshad the same hole,with a worse consequence. An empty column gives zero distinct values and
zero canonical forms, so
collisionsis zero, anddecidereads that asrejected. Rejected claims never reach the written files, so the tool wasreporting a column as consistent when it had measured nothing at all. That
path is
emptynow too.A table count that was really a file count
The generated README could state a table count taken from the number of
files being written.
The write prompt asked for a paragraph about the database and left the count
to the model, while the run summary counted relations from the catalog. Two
counters, and only one of them right.
describeKindsmoves fromcli.tsintoverdict.tsso there is one ofthem.
assembleputs its result onVerifiedasrelations. The promptsays to use that string as given, never to count relations, and never to
report the number of files written as the number of tables.
One thing to check before you take it:
relationscounts what was actuallyexamined, which is not always the whole catalog, because
extractdropsrelations when the time budget runs out or the schema will not fit the
model's input. Rather than let the README overstate the database,
assembleappends the dropped count to the string, so a trimmed run says how many
relations were never looked at. Tell me if you would rather it read
differently.
Testing
The fixture already ships an empty
carstable, so neither case needed newschema. New tests cover the verdict split, the reason being carried through,
the exit code staying put,
assemblehanding the writer a counted string,and the dropped-relation suffix.
50 offline tests pass. A live run against the fixture wrote 13 files and
still described the database by its 11 relations, taken from
relationsrather than counted again.
Two notes on naming, both yours to call. I tried
untestablefirst and itwas too close to
unverifiableto be worth the reader's effort, so thestatus is
empty. AndVerdictcarries the skip reason asskipped,matching
Measurement.skipped, rather thanreason, which already means"why the model believes this claim" on entities and relationships.
I can split this into two commits if you would rather review them separately.