Builds on leapeto/Qwen3-4B-AbstractCoT-warmup
(reproduction of Ramji et al. 2026 warmup at 11h on 2× A100). We ran 800 GRPO
steps on top with verifier reward + length penalty.
MATH-500 results
Stage
maj@8 acc
mean tokens
Leapeto warmup (our probe, n=32, maj@1, T=0.0)
40.6%
337
This model (n=100, maj@8, T=0.6)
52.0%
275
Caveat: empty-z̃ basin
Z̃ length is pegged at 14 tokens; in our probe of the warmup, all 32 problems
emitted the same 14-token abstract sequence (zero mutual information with
prompt). GRPO does not break this — z_first_distinct stays at 1 throughout
800 steps. The model is a compressed-verbal-CoT model with a constant decorative
prefix, not a latent reasoning model.
