Count em-dashes in prose, not in a document's own markup

Docs/CALIBRATION.md

@@ -12,7 +12,7 @@ It is **not an accuracy figure**. Accuracy needs machine-written text to measure
 - **Texts** 296 (452,184 words)
 - **Lengths measured** 649 – 9,328 words (median 832)
 - **Engine** SignsOfAI.Core 0.6.0
-- **Run** 2026-09-01
+- **Run** 2026-09-07
 - **Target false-positive rate** 5%
 
 Every text here was written before 2022 — articles and encyclopedia revisions with a date, and classroom essays from a learner corpus collected years earlier. That is the whole basis for calling it human, and it is a stronger guarantee than any classifier offers about anything. The manifest names each source, its licence and its year, so the claim can be traced rather than trusted.
@@ -41,7 +41,7 @@ A rate that holds in English and fails in Spanish is not one number, and reporti
 
 | Group | Texts | Median | 90th pct | Highest | Threshold for 5% | Best bound it can support |
 |---|---|---|---|---|---|---|
-| **en** | 271 | 8.8 | 18.3 | 33.8 | 30 | 1.4% |
+| **en** | 271 | 8.7 | 18.3 | 33.8 | 30 | 1.4% |
 | **es** | 25 | 7.2 | 15.1 | 18.4 | — | 13.3% |
 
 A dash means this group has too few texts to bound that rate at all — with nothing flagged it still takes roughly seventy-five before the interval alone gets under 5%. That is a statement about the corpus, not the tool.
@@ -54,20 +54,20 @@ The reason the whole exercise exists. If this project cannot show a rate for sec
 |---|---|---|---|---|---|---|
 | **en-anglophone-affiliation** | 21 | 5.9 | 9.0 | 14.2 | — | 15.5% |
 | **en-other-affiliation** | 19 | 6.2 | 13.6 | 18.0 | — | 16.8% |
-| **en-second-language-learner** | 206 | 9.6 | 19.7 | 33.8 | 30 | 1.8% |
+| **en-second-language-learner** | 206 | 9.5 | 19.7 | 33.8 | 30 | 1.8% |
 | **en-wikipedia** | 25 | 4.9 | 10.4 | 23.4 | — | 13.3% |
 | **es-wikipedia** | 25 | 7.2 | 15.1 | 18.4 | — | 13.3% |
 
 A dash means this group has too few texts to bound that rate at all — with nothing flagged it still takes roughly seventy-five before the interval alone gets under 5%. That is a statement about the corpus, not the tool.
 
-Across these groups the median score runs from 9.6 (**en-second-language-learner**) down to 4.9 (**en-wikipedia**), a spread of 4.7 points on a scale of a hundred. The longest tail belongs to **en-second-language-learner** at 19.7 for the ninetieth percentile. At the boundary this page recommends, 30/100, **en-second-language-learner** is flagged 2 of 206 (1%, interval 0.3% – 3.5%); the other 4 groups measured there are flagged nothing at all. It also sits highest in median and ninetieth percentile. One step down, at 25/100, **en-second-language-learner** would be flagged 9 of 206 (4.4%, interval 2.3% – 8.1%) — the only flags anywhere in the corpus at that boundary. That step is why the boundary does not sit at 25. That is the shape of the defect this project criticises, and it is reported here rather than averaged away — smaller than the figures published for other tools, which is a comparison, not an excuse. The groups run from tens of texts to a couple of hundred, and the numbers move as the corpus grows, in whichever direction they move.
+Across these groups the median score runs from 9.5 (**en-second-language-learner**) down to 4.9 (**en-wikipedia**), a spread of 4.6 points on a scale of a hundred. The longest tail belongs to **en-second-language-learner** at 19.7 for the ninetieth percentile. At the boundary this page recommends, 30/100, **en-second-language-learner** is flagged 2 of 206 (1%, interval 0.3% – 3.5%); the other 4 groups measured there are flagged nothing at all. It also sits highest in median and ninetieth percentile. One step down, at 25/100, **en-second-language-learner** would be flagged 9 of 206 (4.4%, interval 2.3% – 8.1%) — the only flags anywhere in the corpus at that boundary. That step is why the boundary does not sit at 25. That is the shape of the defect this project criticises, and it is reported here rather than averaged away — smaller than the figures published for other tools, which is a comparison, not an excuse. The groups run from tens of texts to a couple of hundred, and the numbers move as the corpus grows, in whichever direction they move.
 
 ## Every threshold
 
 | Score at or above | Human texts flagged | Rate | 95% interval |
 |---|---|---|---|
 | 5 | 253 / 296 | 85.5% | 81% – 89% |
-| 10 | 113 / 296 | 38.2% | 32.8% – 43.8% |
+| 10 | 112 / 296 | 37.8% | 32.5% – 43.5% |
 | 15 | 49 / 296 | 16.6% | 12.8% – 21.2% |
 | 20 | 21 / 296 | 7.1% | 4.7% – 10.6% |
 | 25 | 9 / 296 | 3% | 1.6% – 5.7% |

src/SignsOfAI.Core/Analyzers/EmDashAnalyzer.cs

@@ -8,6 +8,9 @@ namespace SignsOfAI.Core.Analyzers;
 /// the tell is density. Emits one document-level finding (like burstiness) when em-dashes appear
 /// far more often than human prose, which averages well under one per 100 words.
 /// Counts real em-dashes (—), horizontal bars (―) and the "--" ASCII stand-in.
+/// Counts them in prose only: a hyphen run inside a fenced block or a code span is a command-line
+/// flag, and a run of three or more is a rule, a table separator or a frontmatter delimiter. None of
+/// those is punctuation, and counting them charged technical documents for their own markup.
 /// Categorised as Rhetorical so the signal counts toward the score, not just the report.
 /// </summary>
 public sealed class EmDashAnalyzer : IAnalyzer
@@ -50,22 +53,66 @@ public IEnumerable<Finding> Analyze(AnalysisContext context)
         };
     }
 
+    /// <summary>
+    /// Counts dashes used as punctuation. Skips fenced code blocks and inline code spans, where a
+    /// "--" is a command-line flag; and skips hyphen runs of three or more, which are separators
+    /// (---, |---|---|, a line of hyphens above a bibliography), never a dash between two clauses.
+    /// The ASCII stand-in for an em-dash is exactly two hyphens.
+    /// </summary>
     private static int CountEmDashes(string text)
     {
         int count = 0;
-        for (int i = 0; i < text.Length; i++)
+        bool insideFence = false;
+
+        foreach (var line in text.Split('\n'))
         {
-            char c = text[i];
-            if (c is '—' or '―') // em dash, horizontal bar
+            var trimmed = line.AsSpan().Trim();
+            if (trimmed.StartsWith("```") || trimmed.StartsWith("~~~"))
             {
-                count++;
+                insideFence = !insideFence;
+                continue;
             }
-            else if (c == '-' && i + 1 < text.Length && text[i + 1] == '-')
+
+            if (!insideFence)
+                count += CountInProseLine(line);
+        }
+
+        return count;
+    }
+
+    private static int CountInProseLine(string line)
+    {
+        int count = 0;
+        for (int i = 0; i < line.Length; i++)
+        {
+            char c = line[i];
+
+            if (c == '`')
+            {
+                // An inline code span holds flags and identifiers, not punctuation. An unmatched
+                // backtick is just a character, so only skip when a closing one exists.
+                int close = line.IndexOf('`', i + 1);
+                if (close < 0)
+                    continue;
+                i = close;
+            }
+            else if (c is '—' or '―') // em dash, horizontal bar
             {
                 count++;
-                i++; // consume the pair, so "---" counts once
+            }
+            else if (c == '-')
+            {
+                int run = 1;
+                while (i + run < line.Length && line[i + run] == '-')
+                    run++;
+
+                if (run == 2)
+                    count++; // the ASCII stand-in; a single hyphen joins words, three or more separate blocks
+
+                i += run - 1;
             }
         }
+
         return count;
     }
 }

src/SignsOfAI.Core/Calibration/published-calibration.json

@@ -1 +1 @@
-{"corpusId":"signsofai-human-baseline","texts":296,"measuredOn":"2026-09-01","engine":"0.6.0","recommendedThreshold":30,"flaggedAtThreshold":2,"rateLow":0.0018549062642001807,"rateHigh":0.024297110573472523,"shortestWords":649,"longestWords":9328,"noisiestRules":[{"ruleId":"lex.just","textShare":0.34797297297297297},{"ruleId":"stat.burstiness","textShare":0.32094594594594594},{"ruleId":"rhet.in-conclusion","textShare":0.30067567567567566},{"ruleId":"rhet.rule-of-three","textShare":0.26013513513513514},{"ruleId":"lex.moreover","textShare":0.22297297297297297},{"ruleId":"rhet.not-only-but","textShare":0.21283783783783783},{"ruleId":"lex.furthermore","textShare":0.1858108108108108},{"ruleId":"lex.crucial","textShare":0.16891891891891891}],"languages":[{"language":"en","texts":271,"recommendedThreshold":30,"bestBound":0.01397699931055571,"rateHighAtThreshold":0.026504623306870336},{"language":"es","texts":25,"recommendedThreshold":null,"bestBound":0.13319225276039096,"rateHighAtThreshold":null}]}
+{"corpusId":"signsofai-human-baseline","texts":296,"measuredOn":"2026-09-07","engine":"0.6.0","recommendedThreshold":30,"flaggedAtThreshold":2,"rateLow":0.0018549062642001807,"rateHigh":0.024297110573472523,"shortestWords":649,"longestWords":9328,"noisiestRules":[{"ruleId":"lex.just","textShare":0.34797297297297297},{"ruleId":"stat.burstiness","textShare":0.32094594594594594},{"ruleId":"rhet.in-conclusion","textShare":0.30067567567567566},{"ruleId":"rhet.rule-of-three","textShare":0.26013513513513514},{"ruleId":"lex.moreover","textShare":0.22297297297297297},{"ruleId":"rhet.not-only-but","textShare":0.21283783783783783},{"ruleId":"lex.furthermore","textShare":0.1858108108108108},{"ruleId":"lex.crucial","textShare":0.16891891891891891}],"languages":[{"language":"en","texts":271,"recommendedThreshold":30,"bestBound":0.01397699931055571,"rateHighAtThreshold":0.026504623306870336},{"language":"es","texts":25,"recommendedThreshold":null,"bestBound":0.13319225276039096,"rateHighAtThreshold":null}]}

tests/SignsOfAI.Core.Tests/EmDashCountingTests.cs

@@ -0,0 +1,122 @@
+using SignsOfAI.Core;
+using SignsOfAI.Core.Model;
+using Xunit;
+
+namespace SignsOfAI.Core.Tests;
+
+/// <summary>
+/// The em-dash signal is a density of punctuation, so it must count punctuation and nothing else.
+/// It used to count every "--" in the raw text, which charged a document for its own markup: a
+/// command-line flag in a code block, a Markdown table separator, a frontmatter delimiter, or the
+/// line of hyphens a student types above a bibliography. Every test here was checked by mutation —
+/// each one fails against the old counter.
+/// </summary>
+public class EmDashCountingTests
+{
+    private readonly AiWritingAnalyzer _analyzer = new();
+
+    // Long enough to clear the 40-word floor, with no dash of any kind in the prose.
+    private const string PlainProse =
+        "The bus was late again on Tuesday, and the shelter had no roof left on its east side. " +
+        "I waited twelve minutes in the rain with a coffee going cold in my hand, then watched it " +
+        "roll past the stop without slowing down. The driver did not look over. I walked the rest " +
+        "of the way to the office and arrived before the next one was due, which tells you " +
+        "something about the timetable and rather more about the traffic on that road.";
+
+    private static Finding? EmDash(AnalysisResult result) =>
+        result.Findings.FirstOrDefault(f => f.RuleId == "rhet.em-dash");
+
+    [Fact]
+    public void Command_line_flags_in_a_fenced_block_are_not_punctuation()
+    {
+        string text = PlainProse + "\n\n```bash\n" +
+            "signsofai check draft.md --json\n" +
+            "signsofai check essay.docx --report out.html\n" +
+            "signsofai baseline a.docx --against b.docx --against c.docx\n" +
+            "```\n";
+
+        Assert.Null(EmDash(_analyzer.Analyze(text, "en")));
+    }
+
+    [Fact]
+    public void A_flag_in_an_inline_code_span_is_not_punctuation()
+    {
+        string text = PlainProse + " Run it with `--json`, then with `--report out.html`, " +
+                      "and compare what `--max-score 40` does to the exit code.";
+
+        Assert.Null(EmDash(_analyzer.Analyze(text, "en")));
+    }
+
+    [Fact]
+    public void A_markdown_table_separator_is_not_punctuation()
+    {
+        string text = PlainProse + "\n\n| Want | Tool | Runs |\n|---|---|---|\n" +
+                      "| A score | the CLI | locally |\n|---|---|---|\n";
+
+        Assert.Null(EmDash(_analyzer.Analyze(text, "en")));
+    }
+
+    [Fact]
+    public void Frontmatter_delimiters_are_not_punctuation()
+    {
+        // Two delimiters alone stay under the three-dash floor; a section break makes three,
+        // which is what a real document with frontmatter and a horizontal rule looks like.
+        string text = "---\nname: signs-of-ai\ndescription: a skill\n---\n\n"
+                      + PlainProse + "\n\n---\n\nAnd the section after the rule, which is also plain prose.";
+
+        Assert.Null(EmDash(_analyzer.Analyze(text, "en")));
+    }
+
+    [Fact]
+    public void A_line_of_hyphens_above_a_bibliography_is_not_punctuation()
+    {
+        // pelic-021970 in the calibration corpus: one separator line typed by a second-language
+        // learner counted as twenty em-dashes and printed "LLMs lean on the em-dash" on their report.
+        string text = PlainProse + "\n\n----------------------------------------\n" +
+                      "1. Dr. Paul Jones, Stress and the Student, 2004.\n";
+
+        Assert.Null(EmDash(_analyzer.Analyze(text, "en")));
+    }
+
+    [Fact]
+    public void The_ascii_stand_in_still_counts_in_prose()
+    {
+        // Exactly two hyphens between words is how people type an em-dash without the key for it.
+        string text = PlainProse.Replace(", and the shelter", " -- and the shelter")
+                                .Replace(", then watched", " -- then watched")
+                                .Replace(", which tells", " -- which tells");
+
+        var finding = EmDash(_analyzer.Analyze(text, "en"));
+        Assert.NotNull(finding);
+    }
+
+    [Fact]
+    public void Real_em_dashes_still_count_in_prose()
+    {
+        string text = PlainProse.Replace(", and the shelter", " — and the shelter")
+                                .Replace(", then watched", " — then watched")
+                                .Replace(", which tells", " — which tells");
+
+        Assert.NotNull(EmDash(_analyzer.Analyze(text, "en")));
+    }
+
+    [Fact]
+    public void Markup_does_not_change_how_many_dashes_are_reported()
+    {
+        // Three real dashes in prose, plus a code block full of flags. The prose alone decides the
+        // count. (The words inside the block still enter the denominator, which understates density
+        // for technical documents — a separate defect, in the shared word count, not in this rule.)
+        string prose = PlainProse.Replace(", and the shelter", " — and the shelter")
+                                 .Replace(", then watched", " — then watched")
+                                 .Replace(", which tells", " — which tells");
+
+        var proseOnly = EmDash(_analyzer.Analyze(prose, "en"));
+        var withMarkup = EmDash(_analyzer.Analyze(
+            prose + "\n\n```bash\nsignsofai check a.md --json --lang en --top 5 --no-color\n```\n", "en"));
+
+        Assert.NotNull(proseOnly);
+        Assert.NotNull(withMarkup);
+        Assert.Contains("(3 in ", proseOnly!.Message);
+        Assert.Contains("(3 in ", withMarkup!.Message);
+    }
+}