LSTM Time Capsule
LSTM Time Capsule is a modern small-compute retest of long-lag credit assignment. Every sequence contains random values and two distant markers; the model must add only the marked values after processing the entire sequence. A parameter-matched vanilla tanh RNN, an LSTM, and a GRU train on identical data and are evaluated at the training length and at longer unseen lengths.
The historical anchor is Hochreiter and Schmidhuber's 1997 Long Short-Term Memory, which motivated LSTM by the difficulty of decaying error flow over extended intervals. This project uses current PyTorch implementations and a new synthetic benchmark; it does not claim to reproduce the paper's original code, exact cells, or tables.
Verified results
All cells trained on the same 16,000 length-100 sequences. The test sets contained 4,000 independently generated sequences at each length.
| Cell | Parameters | Length 100 RMSE | Length 200 RMSE | Length 400 RMSE |
|---|---|---|---|---|
| Vanilla tanh RNN | 4,825 | 0.3855 | 0.4010 | 0.4056 |
| LSTM | 4,641 | 0.0105 | 0.0381 | 0.1050 |
| GRU | 4,357 | 0.0129 | 0.1980 | 0.4264 |
At the training length, LSTM placed 100% of predictions within 0.1 of the target; the vanilla RNN reached 19.03%. At four times the training length, LSTM retained 56.83% within 0.1 while GRU reached 11.50%. This is evidence for these saved checkpoints on this adding task, not a universal architecture ranking.
Reproduce
uv run python projects/lstm-time-capsule/train.py