Your instructions.md notes that the public ARC-AGI-1 evaluation set appears in web-scraped training corpora, so a model's score on it is an upper bound on genuine few-shot rule induction. I measured one specific route by which those tasks travel, and it's one that content-based decontamination doesn't catch.
Across the public ARC corpora, under a subset of the standard augmentation group (D₄ × colour permutation × example ordering): 375 of ARC-AGI-1's 400 evaluation tasks are in the ARC-AGI-2 public training set — verified exhaustively, every one with an explicit group-element witness. Within each corpus separately: zero duplicates beyond exact byte matches.
Byte hashing finds 0 of the 375. JSON-canonical hashing finds 0. That's structural: JCS canonicalizes object key order but not array order, and ARC stores demonstration pairs as a list that is semantically a set. The transformation is pure example reordering.
Augmentation pipelines used for ARC training also include translations — e.g. TRM (arXiv 2512.11847, §3.4) applies colour permutations, dihedral transforms, and translations within bounds. This canonicalizer excludes translations, so the figures here are a lower bound.
Reproduction, ~2 minutes, stdlib only:
git clone https://github.com/maurathat/arc-duplication.git && cd arc-duplication
git clone --depth 1 https://github.com/fchollet/ARC-AGI.git arc
git clone --depth 1 https://github.com/arcprize/ARC-AGI-2.git arc2
python3 measure_cross_corpus.py
I'm not claiming this affects any published score — it measures corpora, not models, and whether it touches a given result depends on what that training run contained.
Has anyone measured overlap against the published augmented ARC datasets? Those are symmetry orbits by construction and don't preserve task ids, so ids can't be the fallback there. That's the case where this method would actually add something over ls.
Method, limits, and the two canonicalizer bugs the property gates caught: https://doi.org/10.5281/zenodo.21940940
Your
instructions.mdnotes that the public ARC-AGI-1 evaluation set appears in web-scraped training corpora, so a model's score on it is an upper bound on genuine few-shot rule induction. I measured one specific route by which those tasks travel, and it's one that content-based decontamination doesn't catch.Across the public ARC corpora, under a subset of the standard augmentation group (D₄ × colour permutation × example ordering): 375 of ARC-AGI-1's 400 evaluation tasks are in the ARC-AGI-2 public training set — verified exhaustively, every one with an explicit group-element witness. Within each corpus separately: zero duplicates beyond exact byte matches.
Byte hashing finds 0 of the 375. JSON-canonical hashing finds 0. That's structural: JCS canonicalizes object key order but not array order, and ARC stores demonstration pairs as a list that is semantically a set. The transformation is pure example reordering.
Augmentation pipelines used for ARC training also include translations — e.g. TRM (arXiv 2512.11847, §3.4) applies colour permutations, dihedral transforms, and translations within bounds. This canonicalizer excludes translations, so the figures here are a lower bound.
Reproduction, ~2 minutes, stdlib only:
I'm not claiming this affects any published score — it measures corpora, not models, and whether it touches a given result depends on what that training run contained.
Has anyone measured overlap against the published augmented ARC datasets? Those are symmetry orbits by construction and don't preserve task ids, so ids can't be the fallback there. That's the case where this method would actually add something over
ls.Method, limits, and the two canonicalizer bugs the property gates caught: https://doi.org/10.5281/zenodo.21940940