Hi - this is a notification, and nothing is being asked of you.
I ran a small pre-registered survey of papers that publish a quantitative figure
about their own verification component, asking whether a third party can
recompute it from the release. AMK qualified on the schedule-IR validator:
across 7,160 adversarial schedules, 6,091 of them unsafe, zero false accepts.
Determination for this repository at commit a514bbc:
RELEASED, NOT COMPUTABLE - on conditions C and A.
paper/results/validator_soundness.json is the most complete aggregate in this
sample: a full confusion matrix, breakdowns by mutant class and by reject
reason, and an explicitly empty false_accepts list. What it does not carry is a
per-schedule record, so the zero cannot be recomputed from primary rows — only
read off a total (C).
Condition A is recorded on your own disclosure, not on my inference: the paper
states that the E2 dynamic oracle shares code with the system, which means the
ground truth is not determinable independently of the thing being judged. It
is in the determination because you wrote it down; most papers in this sample
gave me nothing of the kind to record.
Protocol frozen before collection, every determination with its reason:
https://github.com/kolesnikov-arch/replication-checks/tree/main/survey-verifier-claims
If I have this wrong, say so and I will correct it with attribution. Replies are
published in full and unedited. The window is until 19 August 2026, after which
I stop and do not follow up.
The standard is applied to my own work too: the same repository lists six
defects in my measurement code found during this survey, four of which never
raised an error.
Hi - this is a notification, and nothing is being asked of you.
I ran a small pre-registered survey of papers that publish a quantitative figure
about their own verification component, asking whether a third party can
recompute it from the release. AMK qualified on the schedule-IR validator:
across 7,160 adversarial schedules, 6,091 of them unsafe, zero false accepts.
Determination for this repository at commit a514bbc:
RELEASED, NOT COMPUTABLE - on conditions C and A.
paper/results/validator_soundness.json is the most complete aggregate in this
sample: a full confusion matrix, breakdowns by mutant class and by reject
reason, and an explicitly empty false_accepts list. What it does not carry is a
per-schedule record, so the zero cannot be recomputed from primary rows — only
read off a total (C).
Condition A is recorded on your own disclosure, not on my inference: the paper
states that the E2 dynamic oracle shares code with the system, which means the
ground truth is not determinable independently of the thing being judged. It
is in the determination because you wrote it down; most papers in this sample
gave me nothing of the kind to record.
Protocol frozen before collection, every determination with its reason:
https://github.com/kolesnikov-arch/replication-checks/tree/main/survey-verifier-claims
If I have this wrong, say so and I will correct it with attribution. Replies are
published in full and unedited. The window is until 19 August 2026, after which
I stop and do not follow up.
The standard is applied to my own work too: the same repository lists six
defects in my measurement code found during this survey, four of which never
raised an error.