File size: 2,356 Bytes
f53ead3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | # SAT Retrofit Experiment: Can AR Models Be Forced to Output Multiple Tokens?
## TL;DR
**No.** Autoregressive (AR) models cannot be "snapped" to Semi-Autoregressive (SAT) inference. The hidden states only encode single next-token prediction. Joint AR+SAT training from scratch is required.
## Key Results
### Speed
| Method | Tokens/sec | Speedup |
|--------|-----------|---------|
| AR (baseline) | 75.6 | 1.0x |
| Forced SAT (block=2) | 149.9 | **1.98x** |
### Quality (Perplexity - lower is better)
| Prompt | AR | Forced SAT | Degradation |
|--------|-----|------------|-------------|
| "The quick brown fox" | 16.2 | 424.0 | **26x worse** |
| "In the beginning" | 7.4 | 69.0 | **9x worse** |
| "Once upon a time" | 8.9 | 56.5 | **6x worse** |
| "The scientist discovered" | 10.8 | 296.8 | **27x worse** |
| "Machine learning is" | 7.7 | 133.6 | **17x worse** |
### Example Outputs
**Prompt:** "The scientist discovered that"
- **AR (good):** "the bacteria were able to grow in the air, and that they could also grow in the water."
- **Forced SAT (broken):** "the thatched bacteria were roofing able to offload growling the bacteria spores into into the the"
## Why It Fails
AR model hidden states at position N only encode "what is token N+1" - singular. There is no representation for "what are tokens N+1 AND N+2 simultaneously."
When you force 2-token output:
- Token 1: Predicted from position -1 (correct context) ✓
- Token 2: Predicted from position -2 (STALE context, one step behind) ✗
This creates alternating reasonable/wrong tokens as the model answers questions about outdated context.
## The Solution: Joint AR+SAT Training
Train the model on BOTH objectives from initialization:
- AR loss: predict next token
- SAT loss: predict next N tokens simultaneously
This forces the model to build representations that encode multiple future tokens from each position, enabling both inference modes.
See: [AGILLM-3](https://huggingface.co/OpenTransformer/AGILLM-3-large) for a working implementation.
## Reproducing
```bash
pip install torch transformers
python sat_test.py
```
## Citation
If you use this finding, please cite:
```
@misc{sat_retrofit_2026,
author = {Scott Bisset},
title = {AR Models Cannot Be Retrofitted for SAT Inference},
year = {2026},
publisher = {OpenTransformers Ltd}
}
```
## License
MIT
|