Wave 02: Tension Attention vs Matched Softmax, 22M Pilot
This repository is a self-contained artifact bundle for the Wave 02 controlled head-to-head pilot in the TS Proof Ranker / TensionLM line of work.
It contains both trained arms, all matched seed checkpoints, the tokenizer used for the run, the minimal model source needed to load the checkpoints, training/evaluation scripts, exact configs, JSON receipts, and the aggregate report.
This is not a new proof-ranker ladder release and it is not v5. It is a controlled substrate-comparison pilot.
Research Question
Does sigmoid tension attention beat a matched softmax baseline when parameters, tokens, schedule, optimizer, tokenizer, and seeds are held fixed?
The tested arms were:
tension: sigmoid tension attention with tau-mass normalization.
Mean final validation loss across seeds 11, 23, 37:
tension: 6.287521
softmax: 6.288084
delta loss, tension minus softmax: -0.000563
Mean final validation perplexity across seeds 11, 23, 37:
tension: 537.847747
softmax: 538.130084
delta PPL, tension minus softmax: -0.282337
Tension is numerically lower on the aggregate by a very small margin, but this pilot does not establish a capability edge. The correct reading is outcome_iii_no_capability_edge.
Do not describe this as proved, a breakthrough, or a substrate win.
The FineWeb-Edu parquet shard and generated train/val tensor files are not included because they are large external/reproducible inputs. Their exact provenance and hashes are included below and in reports/fineweb_tokens_meta.json.
Install
Use Python 3.12 if you want to match this receipt closely.
Wave 03 is an interpretability and field-faithfulness receipt over the trained Wave 02 checkpoints. It is not a capability claim and it does not change the Wave 02 verdict.
Protocol:
Extract local token-edge fields from each trained checkpoint.
Select high-field source tokens for each validation sequence.
Replace those source tokens with token id 0.
Compare loss increase against same-count random source-token replacement.
Skip early partial local-window rows so one-neighbor contexts do not dominate the metric.
The key metric is top-random: how much more loss rises when high-field source tokens are ablated than when random source tokens are ablated. Positive values mean the exposed field identifies source tokens that matter more than random under this ablation protocol.
arm
seeds
baseline loss
top increase
random increase
top-random
top>random frac
entropy
top1 share
raw mass
softmax
11, 23, 37
6.248986
0.216230
0.103134
0.113097
0.986111
0.691024
0.441662
1.000000
tension
11, 23, 37
6.247036
0.186365
0.108120
0.078244
0.972222
0.798693
0.300620
3.364441
Delta top-minus-random, tension minus softmax: -0.034852
Conservative interpretation:
Both exposed fields are non-random under this ablation protocol.
Softmax is sharper here: top1 share 0.441662 versus tension 0.300620.
Softmax has the stronger top-token ablation receipt here: top-random 0.113097 versus tension 0.078244.
Tension is more distributed here: normalized entropy 0.798693 versus softmax 0.691024.
Tension exposes raw field mass greater than one (3.364441), unlike softmax's normalized mass of 1.000000.
The safe reading is: tension has an inspectable, distributed edge field, but this pilot does not establish an interpretability advantage over the matched softmax arm.
Matched parameters are required before comparison.
Matched tokenizer, data tensors, optimizer, seed set, batch size, schedule, warmup, and eval cadence are required before comparison.
Any NaN or unmatched arm invalidates the comparison.
A numerically lower validation loss is not, by itself, a substrate capability claim.
This pilot result is best summarized as no clear capability edge.
Known Limitations
This is a 22M-ish pilot, not a large-scale language model result.
It uses three seeds, which is useful as a receipt but still limited.
The run is CPU-produced and slow.
The data sample is one pinned FineWeb-Edu shard transformed into 200,000 fixed-length sequences.
The result probes next-token modeling under this exact schedule; it does not by itself establish general reasoning ability.
The included checkpoints are research artifacts, not production chat/instruction models.
Citation
If you refer to this artifact, cite it as a controlled Wave 02 pilot comparing sigmoid tension attention and matched softmax attention in the TS Proof Ranker / TensionLM repo, with the verdict outcome_iii_no_capability_edge.
