You can deploy this model and start earning money today!

OpenTransformer/
sat-retrofit-experiment

Views

No views yet

SAT Retrofit Experiment: Can AR Models Be Forced to Output Multiple Tokens?

TL;DR

No. Autoregressive (AR) models cannot be "snapped" to Semi-Autoregressive (SAT) inference. The hidden states only encode single next-token prediction. Joint AR+SAT training from scratch is required.

Key Results

Speed

Method	Tokens/sec	Speedup
AR (baseline)	75.6	1.0x
Forced SAT (block=2)	149.9	1.98x

Quality (Perplexity - lower is better)

Prompt	AR	Forced SAT	Degradation
"The quick brown fox"	16.2	424.0	26x worse
"In the beginning"	7.4	69.0	9x worse
"Once upon a time"	8.9	56.5	6x worse
"The scientist discovered"	10.8	296.8	27x worse
"Machine learning is"	7.7	133.6	17x worse

Example Outputs

Prompt: "The scientist discovered that"

AR (good): "the bacteria were able to grow in the air, and that they could also grow in the water."
Forced SAT (broken): "the thatched bacteria were roofing able to offload growling the bacteria spores into into the the"

Why It Fails

AR model hidden states at position N only encode "what is token N+1" - singular. There is no representation for "what are tokens N+1 AND N+2 simultaneously."

When you force 2-token output:

Token 1: Predicted from position -1 (correct context) ✓
Token 2: Predicted from position -2 (STALE context, one step behind) ✗

This creates alternating reasonable/wrong tokens as the model answers questions about outdated context.

The Solution: Joint AR+SAT Training

Train the model on BOTH objectives from initialization:

AR loss: predict next token
SAT loss: predict next N tokens simultaneously

This forces the model to build representations that encode multiple future tokens from each position, enabling both inference modes.

See: AGILLM-3 for a working implementation.

Reproducing

bash

pip install torch transformers
python sat_test.py

Citation

If you use this finding, please cite:

@misc{sat_retrofit_2026,
  author = {Scott Bisset},
  title = {AR Models Cannot Be Retrofitted for SAT Inference},
  year = {2026},
  publisher = {OpenTransformers Ltd}
}

License

MIT

OpenTransformer/sat-retrofit-experiment