Skip to content

Latest commit

 

History

History
105 lines (79 loc) · 5.78 KB

RESULTS.md

File metadata and controls

105 lines (79 loc) · 5.78 KB

AdaSpatial-MLLM final results

Evaluation protocol

All headline accuracy results use greedy decoding, seed 42, batch size 1, a 1,024-token answer cap, and the released Stage-1 checkpoint without training. The 100-example calibration split was used to freeze the adaptive threshold; the 950-example holdout was used once for the final comparison. Accuracy sweeps ran with two workers, so their latency fields are excluded. Clean timing was collected later with one uncontended GPU process.

Main results

Condition n Accuracy Wilson 95% CI Mean latent steps Early exit
fixed4 950 66.00% [62.93, 68.94] 4.00 0.0%
fixed8 950 63.16% [60.04, 66.17] 8.00 0.0%
fixed12 950 67.16% [64.11, 70.07] 11.97 0.0%
adaptive 950 66.95% [63.89, 69.87] 9.69 57.8%

Accuracy and latent steps

Finding 1. Adaptive inference preserves held-out accuracy while reducing latent compute. Adaptive obtained 66.95% versus 67.16% for fixed-12 (-0.21 pp; exact McNemar p=0.935) and reduced mean latent steps by 19.0%.

Task-level accuracy

Condition Among (n=550) Rotation (n=175) Around (n=225)
fixed4 70.91% 48.57% 67.56%
fixed8 68.00% 40.57% 68.89%
fixed12 70.00% 51.43% 72.44%
adaptive 71.64% 49.14% 69.33%

Task-level accuracy

Finding 2. More latent steps are not monotonically better. Fixed-8 reached 40.57% on rotation, below fixed-4 (48.57%) and fixed-12 (51.43%). Latent budget should therefore be measured rather than assumed to improve reasoning.

Exploratory Adaptive+ accuracy routing

After the confirmatory fixed/adaptive experiment, we replayed several training-free routing rules over the frozen prediction matrix. This is explicitly post-hoc exploratory evidence, not a new untouched holdout result.

Policy Accuracy Mean inference latent steps Escalated samples
adaptive 66.95% 9.69 —
fixed12 67.16% 11.97 —
task-aware (Among=adaptive; Rotation/Around=fixed12) 68.11% 10.54 —
task-aware + non-converged 4/8/adaptive vote 68.53% 12.99 194 (20.4%)

The cascade produced 651/950 correct answers, 15 net improvements over adaptive (+1.58 pp, exact McNemar p=0.115) and 13 over fixed12 (+1.37 pp, p=0.275). The task-aware rule without the cascade produced 647/950 correct and used 10.54 mean latent steps, but its differences were also non-significant (p=0.177 vs adaptive; p=0.417 vs fixed12). These results justify an auditable accuracy-oriented prototype and a future confirmatory evaluation; they do not justify claiming a statistically established accuracy gain.

The live API additionally records answer-token confidence and offers an opt-in horizontal-mirror consistency check for directional questions. These signals are exposed as diagnostics and cascade triggers. They have not been calibrated on a fresh test set and are therefore excluded from the 68.53% headline above.

Forced-duration causal controls

The initial controls were confounded because corrupted latents often caused an early model-emitted latent-end token. The corrected controls force exactly 12 latent pads for normal, zero, distribution-matched random, and feature-shuffled states on the same 200 IDs.

Condition n Accuracy Mean steps Δ vs normal McNemar p Mean cosine Relative L2
normal 200 63.50% 12.00 — — — —
zero 200 64.00% 12.00 +0.50 pp 1.000 0.000 1.000
random 200 69.50% 12.00 +6.00 pp 0.126 0.005 1.416
shuffle 200 66.50% 12.00 +3.00 pp 0.512 0.017 1.402

Forced causal controls

Finding 3. Latent corruption does not produce a uniform accuracy collapse. With duration fixed, zero changed accuracy by +0.50 pp (p=1.000), matched-random by +6.00 pp (p=0.126), and feature-shuffle by +3.00 pp (p=0.512). The manipulation checks verify that the states changed; multi-seed replication is still required for random noise.

Clean timing

Condition n Mean latency Median p95 Output tokens Mean steps Peak VRAM
fixed4 50 6.055s 5.712s 8.080s 365.8 4.00 12.64 GiB
fixed8 50 6.165s 5.866s 8.841s 369.9 8.00 12.64 GiB
fixed12 50 6.306s 6.066s 8.189s 372.8 12.00 12.64 GiB
adaptive 50 6.104s 5.829s 8.118s 363.0 10.06 12.64 GiB

Clean latency

Finding 4. Latent-step savings and wall-clock savings are different quantities. Adaptive changed measured generation latency by -3.2% relative to fixed-12 on the clean subset. Answer decoding dominates total generation time, so the 19.0% latent-step reduction should not be reported as a 19.0% speedup.

Limitations

  • Results cover one released 3B Stage-1 checkpoint and MindCube-Tiny; they do not establish generality across checkpoints or benchmarks.
  • The matched-random control uses one deterministic seed. Claims about random interventions require multi-seed replication.
  • Generation latency is dominated by answer decoding, so latent-step reduction need not translate one-for-one into wall-clock speedup.
  • This is a training-free inference study, not a claim of a newly trained foundation model.
  • Adaptive+ routing was designed after the main holdout matrix had been inspected. Its apparent accuracy gain is exploratory and needs a new frozen test set for confirmation.
  • Answer-token confidence is relative over A–D token forms, not a calibrated probability that the spatial answer is factually correct. Mirror consistency is an optional diagnostic, not a proven accuracy improvement.