Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 50 additions & 50 deletions Docs/Calibration/corpus.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion Docs/PARAPHRASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ This page measures what such a rewrite does to a passage, using passages whose a
- **Pair fingerprint** `9fb9e9798f5b2a78`
- **Run** 2026-08-19

Every human half was published before generative models existed, which is the whole basis for calling it human and the same basis the calibration page rests on. Both halves are the same passage at roughly the same length, so the comparison is within a text rather than between two populations: no collection of machine-written prose was assembled, and none was needed. The pairs are cut from the **opening** of each document, which matters and is measured under Length.
Every human half was published before 2022, which is the whole basis for calling it human and the same basis the calibration page rests on. Both halves are the same passage at roughly the same length, so the comparison is within a text rather than between two populations: no collection of machine-written prose was assembled, and none was needed. The pairs are cut from the **opening** of each document, which matters and is measured under Length.

**This half of the study ages, and the human half does not.** A 2019 paper will still have been written in 2019 a decade from now; the rewrite is the work of one model on one day. That is why the model and the date are printed above rather than buried, and why the right response to "but a newer model rewrites differently" is to re-run this with that model rather than to argue about it.

Expand Down
2 changes: 1 addition & 1 deletion src/SignsOfAI.Core/Calibration/PublishedCalibration.cs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ public sealed record PublishedCalibration
/// <summary>The corpus this was measured against, e.g. "signsofai-human-baseline".</summary>
public required string CorpusId { get; init; }

/// <summary>How many texts, all published before generative models could have written them.</summary>
/// <summary>How many texts, all written before 2022, so no generative model could have written them.</summary>
public required int Texts { get; init; }

/// <summary>Date of the measuring run, yyyy-MM-dd.</summary>
Expand Down
2 changes: 1 addition & 1 deletion src/SignsOfAI.Core/Model/AnalysisResult.cs
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public sealed record AnalysisResult

/// <summary>
/// What matched but is not evidence: rules this text uses at a rate measured on writing published
/// before generative models existed. Worth showing — "you use 'furthermore' about as often as
/// written before 2022. Worth showing — "you use 'furthermore' about as often as
/// other people do" is a useful thing to be told — and worth nothing to the score.
/// </summary>
public IReadOnlyList<Finding> Observations => field ??= [.. Findings.Where(f => f.AtHumanRate)];
Expand Down
2 changes: 1 addition & 1 deletion src/SignsOfAI.Core/Model/Finding.cs
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ public sealed record Finding

/// <summary>
/// True when this rule is being used in this text at a rate people write at, measured against
/// writing published before generative models existed. The finding still describes something real
/// writing published before 2022. The finding still describes something real
/// and is still shown, but it is not evidence of a machine and contributes nothing to the score.
///
/// It is marked rather than removed on purpose. Ninety human academic papers produced a median of
Expand Down
2 changes: 1 addition & 1 deletion src/SignsOfAI.Core/Rules/GenreGate.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ namespace SignsOfAI.Core.Rules;
/// <summary>
/// Marks the findings of rules that are describing the genre rather than the machine.
///
/// Measuring the analyzer against ninety texts published before generative models existed produced a
/// Measuring the analyzer against the calibration corpus, all of it written before 2022, produced a
/// number worth staring at: a median of <b>seven</b> flagged tells per human academic paper, 888
/// across the corpus, with only two of the ninety coming back clean. The score survived that — the
/// median was 6.8 out of 100 and nothing reached the recommended threshold — because the rules
Expand Down
2 changes: 1 addition & 1 deletion src/SignsOfAI.Mcp/Tools/ReportTools.cs
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ an academic-integrity committee — the finished artefact rather than a summary
disagree with its own bibliography. Checkable facts are named at the top and kept apart from the score,
which is an opinion about prose.
Every report prints how often this build is wrong, measured for the language actually analysed against
texts published before generative models existed, and names the rules known to fire on human writing so
texts written before 2022, and names the rules known to fire on human writing so
the reader can weigh evidence that leans on one. Below the threshold that measurement supports, no
verdict is given at all.
Runs FULLY OFFLINE. The result contains material from the document, so treat it as you would the
Expand Down
195 changes: 114 additions & 81 deletions src/SignsOfAI.Web/wwwroot/why.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/SignsOfAI.Core.Tests/ChatResidueTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ namespace SignsOfAI.Core.Tests;
///
/// That is also why this batch was admissible when most of the pattern set it came from was not.
/// The rules were screened against the calibration corpus first — 249,455 words of English and
/// 39,712 of Spanish, all published before generative models existed — and every one of them scored
/// 39,712 of Spanish, all published before 2022 — and every one of them scored
/// zero. Re-running the calibration afterwards left the published false-positive rate untouched,
/// which is the point: they cost nothing to carry.
///
Expand Down
137 changes: 137 additions & 0 deletions tests/SignsOfAI.Core.Tests/WhyPageTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
using System.Text.RegularExpressions;
using SignsOfAI.Core.Calibration;
using Xunit;

namespace SignsOfAI.Core.Tests;

/// <summary>
/// <c>why.html</c> is the page README line 15 sends teachers to first, and it has no build step
/// behind it on purpose — it must open from a downloads folder years from now on a machine that
/// has never heard of this project. The cost of that choice is that its figures are typed by hand,
/// and a comment asking the next person to remember is not a guarantee.
///
/// It was not. The page sat from 5 August through two corpus changes still showing "ninety texts"
/// and a green zero, while the project's own published rate had stopped being zero — on the one
/// page whose entire argument is that this project tells you how often it is wrong.
///
/// So these tests read the same <c>published-calibration.json</c> the engine ships and require the
/// page to agree with it. When the calibration moves, this fails by name and says what to redraw.
/// </summary>
public class WhyPageTests
{
private static readonly string Page = File.ReadAllText(PagePath());

/// <summary>The snapshot the engine ships. A build without one is a broken build, not a pass.</summary>
private static PublishedCalibration Shipped =>
PublishedCalibration.Current ?? throw new InvalidOperationException(
"The build carries no published calibration, so why.html cannot be checked against it.");

private static string PagePath()
{
var dir = new DirectoryInfo(AppContext.BaseDirectory);
while (dir is not null && !Directory.Exists(Path.Combine(dir.FullName, "src")))
dir = dir.Parent;

Assert.NotNull(dir);
return Path.Combine(dir!.FullName, "src", "SignsOfAI.Web", "wwwroot", "why.html");
}

// The page writes Spanish figures with a comma and English ones with a dot, so every check
// wants both spellings and both must be present — one language quietly going stale is the
// failure this guards.
private static void AssertBothLanguagesSay(string english, string spanish, string what)
{
Assert.True(Page.Contains(english, StringComparison.Ordinal),
$"why.html no longer states {what} in English ({english}). The calibration moved; redraw the page.");
Assert.True(Page.Contains(spanish, StringComparison.Ordinal),
$"why.html no longer states {what} in Spanish ({spanish}). The calibration moved; redraw the page.");
}

[Fact]
public void The_corpus_size_on_the_page_is_the_corpus_size_that_shipped()
{
int texts = Shipped.Texts;

AssertBothLanguagesSay($"{texts} HUMAN TEXTS", $"{texts} TEXTOS HUMANOS", "the corpus size in its chart");
AssertBothLanguagesSay($"{texts} texts written before", $"{texts} textos escritos antes", "the corpus size in its prose");
}

[Fact]
public void The_dot_chart_draws_one_circle_per_text_and_fills_the_flagged_ones()
{
// Both halves of the page carry the same chart, so every count here is doubled.
int texts = Shipped.Texts;
int flagged = Shipped.FlaggedAtThreshold;

int circles = Regex.Matches(Page, @"<circle cx=""\d+"" cy=""\d+"" r=""3\.2""").Count;
int filled = Regex.Matches(Page, @"<circle cx=""\d+"" cy=""\d+"" r=""3\.2"" fill=""#f59e0b""").Count;

Assert.Equal(texts * 2, circles);
Assert.Equal(flagged * 2, filled);
}

[Fact]
public void The_upper_bound_drawn_is_the_upper_bound_published()
{
// The bound the page may claim is the one at the recommended threshold, never a better one
// measured at a stricter cut. Printing the best bound beside the recommended threshold is
// the defect the committee caught in the evidence report on 1 September.
string en = (Shipped.RateHigh * 100).ToString("0.0");
AssertBothLanguagesSay($"less than {en}%", $"menos del {en.Replace('.', ',')} %", "the upper bound of its interval");
}

[Fact]
public void The_measured_rate_drawn_is_the_measured_rate_published()
{
var c = Shipped;
string rate = (c.FlaggedAtThreshold / (double)c.Texts * 100).ToString("0.0");

AssertBothLanguagesSay($">{rate}%<", $">{rate.Replace('.', ',')} %<", "the rate it actually measured");
}

[Fact]
public void The_english_bar_shows_the_bound_at_the_threshold_not_the_best_one()
{
var english = Shipped.Languages.Single(l => l.Language == "en");
Assert.NotNull(english.RateHighAtThreshold);

string atThreshold = (english.RateHighAtThreshold!.Value * 100).ToString("0.0");
string best = (english.BestBound * 100).ToString("0.0");

AssertBothLanguagesSay($">{atThreshold}%<", $">{atThreshold.Replace('.', ',')} %<",
"English's bound at the recommended threshold");

// And it must not quietly show the flattering one instead.
Assert.DoesNotContain($"font-weight=\"700\">{best}%<", Page, StringComparison.Ordinal);
}

[Fact]
public void The_page_names_the_engine_that_produced_its_figures()
{
string engine = Shipped.Engine;

AssertBothLanguagesSay($"engine {engine}", $"motor {engine}", "the engine version its figures came from");
}

[Fact]
public void Nothing_on_the_page_still_claims_a_rate_of_zero()
{
// The claim that survived six days after it stopped being true.
Assert.DoesNotContain("zero out of ninety", Page, StringComparison.OrdinalIgnoreCase);
Assert.DoesNotContain("cero de noventa", Page, StringComparison.OrdinalIgnoreCase);
}

[Fact]
public void The_page_still_asks_nothing_of_any_server()
{
// Its own first rule: a page arguing that student work never leaves your machine cannot
// itself fetch from somebody else's. Guarded here because a redraw is when it would slip.
Assert.DoesNotContain("<script", Page, StringComparison.OrdinalIgnoreCase);
foreach (Match m in Regex.Matches(Page, @"(?:src|href)=""(https?:)?//([^""]+)""", RegexOptions.IgnoreCase))
{
string host = m.Groups[2].Value;
bool allowed = host.StartsWith("github.com/peopleworks/", StringComparison.OrdinalIgnoreCase);
Assert.True(allowed, $"why.html would fetch from {host}; it must ask nothing of any server.");
}
}
}
8 changes: 4 additions & 4 deletions tools/SignsOfAI.Calibration/Fetch.cs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ namespace SignsOfAI.Calibration;
/// Assembles a corpus of writing that is known to be human.
///
/// "Known" is doing real work here, and it rests on one fact rather than on a classifier: every text
/// collected was published before generative models could have written it. A paper with a 2019 DOI
/// collected was published before 2022, so no generative model could have written it. A 2019 DOI
/// and a Wikipedia revision stamped 2021 were not produced by something that did not exist. No
/// detector can offer a guarantee that strong about anything, which is precisely why the corpus has
/// to come from dates and not from judgement.
Expand Down Expand Up @@ -87,7 +87,7 @@ private static async Task<string> GetAsync(string url, int attempts = 4)
];

/// <summary>
/// Open-access research articles published well before generative models, from PLOS.
/// Open-access research articles published well before 2022, from PLOS.
///
/// Academic prose is not a student essay and the report says so. It is, however, the closest
/// freely licensed writing to what a teacher actually reads, and it comes with author
Expand Down Expand Up @@ -194,7 +194,7 @@ private static string ExtractJats(string xml)
// ---- Wikipedia -------------------------------------------------------------------------------

/// <summary>
/// Article prose as it stood before generative models, taken from the revision history.
/// Article prose as it stood before 2022, taken from the revision history.
///
/// This is the only route found that yields substantial Spanish prose that is provably pre-model,
/// openly licensed and machine-fetchable — and Spanish is the half of this project that nobody
Expand Down Expand Up @@ -261,7 +261,7 @@ public static async Task<List<CorpusEntry>> WikipediaAsync(
Url = $"https://{language}.wikipedia.org/w/index.php?oldid={revid}",
File = file,
Sha256 = CorpusManifest.HashText(text),
Note = $"Revision of \"{title}\" as of {stamp}, before generative models.",
Note = $"Revision of \"{title}\" as of {stamp}, before 2022.",
});

Console.WriteLine($" {entries.Count,3}. {id} {stamp[..10]} {CountWords(text),6:N0} words {Shorten(title)}");
Expand Down
2 changes: 1 addition & 1 deletion tools/SignsOfAI.Calibration/ParaphraseReport.cs
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ public static string Render(
sb.AppendLine($"- **Run** {generatedOn}");
sb.AppendLine();
sb.AppendLine(
"Every human half was published before generative models existed, which is the whole basis " +
"Every human half was published before 2022, which is the whole basis " +
"for calling it human and the same basis the calibration page rests on. Both halves are the " +
"same passage at roughly the same length, so the comparison is within a text rather than " +
"between two populations: no collection of machine-written prose was assembled, and none " +
Expand Down
Loading