SAT Retrofit Experiment: Can AR Models Be Forced to Output Multiple Tokens?
TL;DR
No. Autoregressive (AR) models cannot be "snapped" to Semi-Autoregressive (SAT) inference. The hidden states only encode single next-token prediction. Joint AR+SAT training from scratch is required.
Key Results
Speed
| Method | Tokens/sec | Speedup |
|---|---|---|
| AR (baseline) | 75.6 | 1.0x |
| Forced SAT (block=2) | 149.9 | 1.98x |
Quality (Perplexity - lower is better)
| Prompt | AR | Forced SAT | Degradation |
|---|---|---|---|
| "The quick brown fox" | 16.2 | 424.0 | 26x worse |
| "In the beginning" | 7.4 | 69.0 | 9x worse |
| "Once upon a time" | 8.9 | 56.5 | 6x worse |
| "The scientist discovered" | 10.8 | 296.8 | 27x worse |
| "Machine learning is" | 7.7 | 133.6 | 17x worse |
Example Outputs
Prompt: "The scientist discovered that"
- AR (good): "the bacteria were able to grow in the air, and that they could also grow in the water."
- Forced SAT (broken): "the thatched bacteria were roofing able to offload growling the bacteria spores into into the the"
Why It Fails
AR model hidden states at position N only encode "what is token N+1" - singular. There is no representation for "what are tokens N+1 AND N+2 simultaneously."
When you force 2-token output:
- Token 1: Predicted from position -1 (correct context) ✓
- Token 2: Predicted from position -2 (STALE context, one step behind) ✗
This creates alternating reasonable/wrong tokens as the model answers questions about outdated context.
The Solution: Joint AR+SAT Training
Train the model on BOTH objectives from initialization:
- AR loss: predict next token
- SAT loss: predict next N tokens simultaneously
This forces the model to build representations that encode multiple future tokens from each position, enabling both inference modes.
See: AGILLM-3 for a working implementation.
Reproducing
pip install torch transformers
python sat_test.py
Citation
If you use this finding, please cite:
@misc{sat_retrofit_2026,
author = {Scott Bisset},
title = {AR Models Cannot Be Retrofitted for SAT Inference},
year = {2026},
publisher = {OpenTransformers Ltd}
}
License
MIT