feat: path-patching DE/TE on the native head — DE–TE scatter (#44, Part 1)

[xiaohan2012]

Part 1 of #44 — the DE–TE scatter (Hydra Fig 2c analog), plus a per-row extension. Additive: touches no existing module.

What & why

The scatter needs two rulers in one basis. This adds both, read through the model's own decoder (not per-layer tuned decoders) so they're comparable by construction.

• DE(m) — direct effect, path-patching (method B). Residual is additive ⇒ freezing downstream at clean is arithmetic, not a per-layer forward:

  r_L^DE(m) = r_L^clean − a_m + ã_m
  

  One clean forward captures every a_m; DE(m) for all m is residual arithmetic through the true decoder (final LN σ recomputed) — not a fixed û on one layer (that would be method A).

• TE(m) — total effect. Ablate-and-react (inject_delta), read the final residual through the same decoder.

• DE and TE share one donor draw per (m, donor) → clean CE pairing for Part 2.

• ã_m: resample — role-matched donor δ, metric averaged (reusing Ablation method: resample (cross-table) instead of zero/skip #35).

• Per-row (--per-row): also store un-aggregated per-test-row DE/TE → closes the row-level sign-cancellation loophole (E1 robustness). Same forwards, just skip the mean/median.

Files

• src/tfm_lens/evaluation/direct_effect.py — direct_total_effect (+ per_row) + final_decoder_logits (the shared ruler).
• src/tfm_lens/evaluation/layerwise.py — _gt_logit → public gt_logit (now used cross-module).
• scripts/run_direct_effect_sweep.py — per-task DE/TE + native-final normalizer → out/de_<model>.json (--per-row optional).
• scripts/plot_de_te_scatter.py — D1: DE(x) vs TE(y) + diagonal, per (layer, dataset), z-scored per task; redundant DE≈0 stripe flagged.
• scripts/plot_de_te_perrow.py — per-row hexbin + CE histogram.
• tests/test_direct_effect.py.

Tests

• Headline invariant: the O(1)-forward arithmetic DE == a real frozen-downstream forward (skip m, pin every downstream layer to its clean contribution). Parametrized over 3D / 4D / double-stream — if these match, the shortcut is exact for any decoder/layout.
• DE == logit-lens closed form on a linear head (method A == B).
• Per-row: shape + aggregate DE == mean(per-row DE) on the linear coordinate.
• resample path shape/keys/finiteness; missing-donor guard.
• Full suite: 109 passed, 8 skipped (skips pre-existing: einx absent locally).

Results

Full 4-model × 15-task run + analysis → no self-repair (falsification of #44). See the results comment.

Not in this PR (per #44)

• Part 2: CE decomposition + CE~DE regression (independent CE).
• Part 3: LN-artifact (RQ3) fold-σ subtraction.

🤖 Generated with Claude Code

[xiaohan2012]

Left some comments.

Other remarks:

• add type hints to improve readability.
• perhaps move some common util functions to some utility script

[xiaohan2012]

Review pass in df5077f. Summary:

• Renames / clarity: native_head_logits→final_decoder_logits, _swap→patched_residual, descriptive vars (clean_contributions / clean_residual / replacement), "head"→"decoder" throughout. a_m documented = clean write r_m − r_{m-1}.
• No zero ablation: removed from module, sweep, and tests — resample only.
• Dedup / utils: clone_residual → utils.py (was self_repair._clone_residual); native_final_logits factored out of native_final_auc so TE reuses it (deleted the bespoke _te_forward).
• Type hints: added everywhere, incl. Residual / Coords aliases.

Tests rewritten to the primitive invariant with an arbitrary ã_m (no zero): patched_residual == frozen-downstream forward (3D/4D/double-stream) + linear-head A==B. Full suite green; re-smoked on LimiX-2M end-to-end.

[xiaohan2012]

#44 Part 1 — result: no self-repair (path-patching DE vs TE · 4 models)

What would show self-repair — 3 evidence, all required:

	evidence	signature
E1	decoupling at real DE	below-diagonal cluster (TE≪DE) with large DE (not the DE≈0 stripe)
E2	systematic law	tight CE~DE line (Hydra R² 0.92, slope 0.69)
E3	active, not artifact	localizable to layers + survives LN-σ subtraction

Logic: exists → mechanism → self-repair.

Reference — what self-repair looks like (Hydra, McGrath 2023)

E1 · below-diagonal downstream-repair mass (Δ_ablate vs Δ_unembed):

E2 · tight compensation law — Layer 23, R²=0.92, slope=0.69:

---

E1 — absent. DE–TE scatter (task×layer, margin, z-scored):

• 70–90% of points in the DE≈0 redundant stripe (non-redundant: LimiX 55/180, Mitra 28/180, TabICL 34/180, TabFM 39/360) → no large-DE to decouple.
• rest: mean CE ≈ 0σ (+0.09 / +0.01 / −0.08 / +0.00); below-diag% noisy 35–68%, no direction.

E1 robustness — is the null hidden by row-averaging? (per-row → no)

Aggregate collapses ~500 rows/point → a null CE could hide a signal. The per-row CE distribution shape disambiguates the hypotheses:

per-row CE shape	hypothesis	self-repair?
single peak at 0	genuinely flat — no compensation	no (honest null)
bimodal / two fat tails	sign cancellation (½ CE>0 repair, ½ CE<0 break)	would hide repair
right-skew / fat +tail	diluted subpopulation (repair on a few hard rows)	would hide repair
+mean, unimodal	uniform repair	yes
left-skew / −mean	net breakage / amplification	opposite of repair

• observed: single peak at 0 for all 4 → genuinely flat, not cancellation, not a diluted subpopulation.
• per-task CE / 15 tables: LimiX +0.06±0.14 (12/15 pos, but 8/15 on gt_logit → not coord-stable), Mitra −0.12 (3/15), TabICL −0.23 (1/15), TabFM +0.00 (10/15).
• only stable signal = TabICL CE<0 (left-lean, amplification), opposite of repair.

---

E2 — absent. Hydra's claim = a systematic per-layer law (Fig-4d):

• CE~DE regression R² + slope form a band across middle-late layers.
• apex at L23: R²=0.92, slope=0.69.

TFM analog — R² + slope vs depth, task-aggregated (15 independent pts/layer):

model	apex (peak R²)	slope	R²
LimiX-2M	L10	−0.03	0.00
Mitra	L10	+0.08	0.04
TabICLv2	L10	−0.41	0.58 (negative = amplification)
TabFM	L18	−0.08	0.04

• no layer has both R²→0.92 and slope∈(0,1). Most layers are redundant (DE≈0, hollow markers).
• the one structured layer (TabICL L10, R²0.58) has a negative slope = downstream amplification, opposite of repair.
• per-row regression (Hydra's per-prompt analog, thousands of pts) agrees — apex layers negative/weak either way.
• CE=DE−TE is method-A (mechanical coupling → biased toward a positive law); even so, no Hydra band. Coupling-free version = independent CE (Part 2).

single-layer view — apex DE vs CE scatter (Hydra Fig-4b analog)

Each model's peak-R² layer (final layer excluded — no downstream ⇒ CE≡0).

• Hydra L23 = tight positive cloud below the 1:1 line.
• ours = flat blobs (R²≈0), or TabICL's downward line (amplification).

---

E3 — not reached, moot.

• E1/E2 absent → nothing to attribute/artifact-correct.
• LN branch pre-empted by Mitra (linear head, ε=0, also null) → the null isn't LN masking a real effect.
• per-layer CE + σ-freeze = remaining RQ3/Part-2; only if a signal appears.

---

Takeaway:

• 4 architectures × 2 resolutions × Mitra linear anchor → no self-repair.
• TFMs are redundant (DE≈0 dispensable layers), not actively self-repairing like LLMs.
• Falsification is the Transfer LLM self-repair to TFMs — 3-part plan (path-patching DE → CE regression → LN artifact) #44 result.

[xiaohan2012]

scripts/plot_de_te_scatter.py and scripts/plot_de_te_perrow.py?

这两个脚本什么关系 ?

[xiaohan2012]

name of this file should be updated, it is not self-repair as we discovered, let's find out a proper name

[xiaohan2012]

is self_repair still correct? what does this script do?

[xiaohan2012]

Does dip→recover mean self-repair? (a note on balef2026's method)

Ablate a layer, decode every depth with its tuned probe: the answer dips right after the ablated layer, then recovers to the no-ablation baseline by the final layer.

Looks like the model repairs itself. Does it? — not from this measurement. The gap:

What self-repair is

Def (McGrath 2023, Hydra Effect): ablate a component → downstream actively compensates → net damage ≪ the component's own importance.

Make "compensate" measurable — two counterfactuals for ablating layer m:

• DE (direct effect) — remove m's write, freeze downstream at clean. What m directly contributes (damage if nobody reacts).
• TE (total effect) — remove m, let downstream react. The net damage.
• CE = DE − TE — what downstream made up. self-repair ⟺ CE > 0 (TE ≪ DE).

Why you need both — one number can't express it:

• TE alone is ambiguous. TE≈0 = layer never mattered (redundant) or mattered and was fully repaired. Same reading, opposite mechanisms.
• DE alone is ambiguous. DE>0 = load-bearing (TE=DE) or repaired (TE≪DE).
• Only the gap DE − TE isolates compensation — "downstream compensated" is "direct importance > net importance", a two-quantity comparison.

Mediation view: DE = direct path m→out, TE = total, CE = indirect path through downstream. Self-repair = the mediator offsetting the direct damage.

balef2026 measures only TE

• dip→recover is a per-depth decoded trajectory. Its "recover to baseline" is TE≈0 — net damage at the final layer, downstream already free to react.
• It never freezes downstream → never produces DE.
• So it reads exactly TE≈0 — the ambiguous case — as repair. But TE≈0 is equally redundancy; only DE separates them.

	DE	TE	verdict
balef's "recover"	not measured	≈0	redundant or repaired — undecidable

Not that the conclusion is wrong — the method can't form the quantity that defines self-repair. Studying it requires the frozen-downstream DE. The paper stops where the question begins.

Two further confounds in the tuned-decoder read

Even granting a signal:

1. Decodability ≠ causal contribution. A per-depth tuned probe measures how well the answer can be decoded from the residual — info present, not info used by the native readout. And each depth uses a different probe → dip and recover are read on different rulers.
2. dip→recover ⊂ passive redundancy. If the answer is already duplicated in the residual, deleting one copy dips a local probe but the others survive → the final readout still decodes it. Nobody rewrote anything; later layers pick up redundant info. (Cheap in TFMs — the input table stays in context.)

Measuring DE and TE on one ruler — and what it shows — is in the results comment.

[xiaohan2012]

is it a good pattern to name a script with prefix underscore? perhaps put those into tfm_lens utils or plot_utils, etc