| # Recurrent depth × context: causal-v4 exploratory experiment |
|
|
| Date: 2026-10-09 UTC |
|
|
| Question: Does recurrent depth amplify the benefit of longer context at tiny scale? |
|
|
| ## Reproducibility |
|
|
| Source: `recurrent_ctx_interaction_v4_causal.py` |
| Source SHA-256: `d5ef19a4cc0a6a26137865553e46a707882dae111ddd52832bd826f052dc0d09` |
| Results SHA-256: `89acd1b3f005cdcf35f33a82bd1f194d9c3046a79c3531a7fdabd5b244b8ad75` |
| Correctness preflight: passed (target alignment, causal attention, gradients, memory, checkpoint reload). |
| Tracked execution: exit code 0; duration 61.64 seconds; no recorded stderr. |
|
|
| Byte-level TinyStories next-token prediction. A 95/5 contiguous raw-data split separates training from validation, with source-code caps of 50 million training bytes and 2 million validation bytes. Four block applications; d_model 256; four attention heads; vocabulary 256; tied token embeddings; AdamW; 2,000 updates; batch size 32; learning rate 0.0003; seed 42. Recurrent feed-forward width 2048; stacked width 128. Training uses random sampled sequences. Validation is performed in eval mode without gradients. |
| |
| ## Recorded metrics |
| |
| | Configuration | Parameters | Context | Training-token positions | Final validation loss | Final validation perplexity | |
| |---|---:|---:|---:|---:|---:| |
| | recurrent_ctx64 | 1,397,504 | 64 | 4,096,000 | 1.80549 | 6.08295 | |
| | recurrent_ctx256 | 1,446,656 | 256 | 16,384,000 | 2.23559 | 9.35204 | |
| | stacked_ctx64 | 1,402,880 | 64 | 4,096,000 | 1.92167 | 6.83238 | |
| | stacked_ctx256 | 1,452,032 | 256 | 16,384,000 | 2.27253 | 9.70391 | |
| |
| ## Interpretation and limitations |
| |
| Recurrent configurations achieved lower validation perplexity at both context lengths in this particular run. The recurrent-minus-stacked advantage in perplexity was approximately 0.749 at context 64 and 0.352 at context 256. This is exploratory evidence, not proof of an architecture benefit or a recurrence × context interaction. Context conditions processed different training-token counts (4.096M vs 16.384M) and were evaluated at different context lengths. Recurrent and stacked variants use different feed-forward widths to approximate parameter matching, not identical architectures. The random seed was initialized once, not reset per configuration. Validation means are unweighted averages of batch losses; the last partial batch can have disproportionate influence. Only one seed was tested. No inference-speed or broad quality evaluation is implied. Previous noncausal v3 results are invalid because they permitted future-token leakage. |
| |
| A stronger follow-up would match training-token exposure, evaluate all conditions on an identical held-out target set using comparable available histories, use multiple independent seeds, and report confidence intervals. |
| |
