# SAT Retrofit Experiment: Can AR Models Be Forced to Output Multiple Tokens?

## TL;DR
**No.** Autoregressive (AR) models cannot be "snapped" to Semi-Autoregressive (SAT) inference. The hidden states only encode single next-token prediction. Joint AR+SAT training from scratch is required.

## Key Results

### Speed
| Method | Tokens/sec | Speedup |
|--------|-----------|---------|
| AR (baseline) | 75.6 | 1.0x |
| Forced SAT (block=2) | 149.9 | **1.98x** |

### Quality (Perplexity - lower is better)
| Prompt | AR | Forced SAT | Degradation |
|--------|-----|------------|-------------|
| "The quick brown fox" | 16.2 | 424.0 | **26x worse** |
| "In the beginning" | 7.4 | 69.0 | **9x worse** |
| "Once upon a time" | 8.9 | 56.5 | **6x worse** |
| "The scientist discovered" | 10.8 | 296.8 | **27x worse** |
| "Machine learning is" | 7.7 | 133.6 | **17x worse** |

### Example Outputs

**Prompt:** "The scientist discovered that"

- **AR (good):** "the bacteria were able to grow in the air, and that they could also grow in the water."
- **Forced SAT (broken):** "the thatched bacteria were roofing able to offload growling the bacteria spores into into the the"

## Why It Fails

AR model hidden states at position N only encode "what is token N+1" - singular. There is no representation for "what are tokens N+1 AND N+2 simultaneously."

When you force 2-token output:
- Token 1: Predicted from position -1 (correct context) ✓
- Token 2: Predicted from position -2 (STALE context, one step behind) ✗

This creates alternating reasonable/wrong tokens as the model answers questions about outdated context.

## The Solution: Joint AR+SAT Training

Train the model on BOTH objectives from initialization:
- AR loss: predict next token
- SAT loss: predict next N tokens simultaneously

This forces the model to build representations that encode multiple future tokens from each position, enabling both inference modes.

See: [AGILLM-3](https://huggingface.co/OpenTransformer/AGILLM-3-large) for a working implementation.

## Reproducing

```bash
pip install torch transformers
python sat_test.py
```

## Citation

If you use this finding, please cite:
```
@misc{sat_retrofit_2026,
  author = {Scott Bisset},
  title = {AR Models Cannot Be Retrofitted for SAT Inference},
  year = {2026},
  publisher = {OpenTransformers Ltd}
}
```

## License
MIT