Skip to content

Separate empty claims from failed measurements, and count relations once - #1

Merged
FilipKalcic1 merged 1 commit into
FilipKalcic1:mainfrom
stepolan:empty-verdicts-and-relation-count
Sep 19, 2026
Merged

FilipKalcic1 merged 1 commit into
FilipKalcic1:mainfrom
stepolan:empty-verdicts-and-relation-count

Conversation

@stepolan

Copy link
Copy Markdown
Contributor

The two fixes you agreed to take. One commit, both tested against the fixture.

"Unverifiable" was hiding two different outcomes

When a claim is measured against a table with no rows, nothing failed. There
was nothing to measure, and the claim came out neither supported nor refuted.

Calling that unverifiable reads like a measurement the tool could not take.
On a schema with many empty or lightly used tables it fills the output with
what look like failures, and buries the claims that really are still open.

A measurement that finds nothing to measure is now marked empty and gets
its own verdict. Timeouts, exhausted budgets, SQL errors and claims naming a
table that does not exist stay unverifiable, because those are genuinely
unresolved.

Verdicts carry the reason now, so the summary collapses them into one line
grouped by reason. From a fixture run:

empty: 1 claim with nothing to measure (1 a materialized view that has never been refreshed cannot be read)

The write prompt says an empty claim is not a finding and must not be
listed one by one. Exit codes are untouched.

While separating the two I found inconsistent_values had the same hole,
with a worse consequence. An empty column gives zero distinct values and
zero canonical forms, so collisions is zero, and decide reads that as
rejected. Rejected claims never reach the written files, so the tool was
reporting a column as consistent when it had measured nothing at all. That
path is empty now too.

A table count that was really a file count

The generated README could state a table count taken from the number of
files being written.

The write prompt asked for a paragraph about the database and left the count
to the model, while the run summary counted relations from the catalog. Two
counters, and only one of them right.

describeKinds moves from cli.ts into verdict.ts so there is one of
them. assemble puts its result on Verified as relations. The prompt
says to use that string as given, never to count relations, and never to
report the number of files written as the number of tables.

One thing to check before you take it: relations counts what was actually
examined, which is not always the whole catalog, because extract drops
relations when the time budget runs out or the schema will not fit the
model's input. Rather than let the README overstate the database, assemble
appends the dropped count to the string, so a trimmed run says how many
relations were never looked at. Tell me if you would rather it read
differently.

Testing

The fixture already ships an empty cars table, so neither case needed new
schema. New tests cover the verdict split, the reason being carried through,
the exit code staying put, assemble handing the writer a counted string,
and the dropped-relation suffix.

50 offline tests pass. A live run against the fixture wrote 13 files and
still described the database by its 11 relations, taken from relations
rather than counted again.

Two notes on naming, both yours to call. I tried untestable first and it
was too close to unverifiable to be worth the reader's effort, so the
status is empty. And Verdict carries the skip reason as skipped,
matching Measurement.skipped, rather than reason, which already means
"why the model believes this claim" on entities and relationships.

I can split this into two commits if you would rather review them separately.

A claim measured against a table with no rows has not failed. There was
nothing to measure, and the claim came out neither supported nor refuted.
Reporting that as "unverifiable" reads as a measurement that could not be
taken, and it buries the claims that really are still open.

A measurement that finds nothing to measure is now marked empty and gets its
own verdict. Timeouts, exhausted budgets, SQL errors and claims naming a
table that does not exist stay unverifiable. Verdicts carry the reason, so
the summary collapses the empty ones into one line grouped by reason, and
the write prompt says an empty claim is not a finding and must not be listed
one by one. Exit codes do not move.

inconsistent_values had the same hole and a worse consequence. An empty
column gives zero distinct values and zero canonical forms, so collisions is
zero, and decide read that as "rejected". Rejected claims never reach the
written files, so the tool was reporting a column as consistent when it had
measured nothing at all.

Second: the generated README could state a table count taken from the number
of files being written. The write prompt asked for a paragraph about the
database and left the count to the model, while the run summary counted
relations from the catalog.

describeKinds moves from cli.ts into verdict.ts, assemble puts its result on
Verified as "relations", and the prompt says to use that string as given and
never to count relations itself. Relations dropped to fit the budget or the
model's input limit are counted on the end of that string, so the README
never claims more than was examined.

Tested against the fixture: the empty cars table gives a relationship claim
and an inconsistent_values suspicion that both come back empty. A live run
wrote 13 files and still described the database by its 11 relations. 50
offline tests pass.
@FilipKalcic1
FilipKalcic1 merged commit b0fff94 into FilipKalcic1:main Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants