Recurrent depth × context length: a bounded synthetic comparison
Question
Does recurrent depth improve the benefit of longer context at tiny scale?
Setup
This report summarizes the completed recurrent_ctx_interaction_v4_causal run (exit 0; 63.2 seconds), recorded in recurrent_ctx_interaction_results.json. Four configurations were trained for 2,000 steps, with d_model=256, 4 heads, byte vocabulary 256, batch size 32 and learning rate 3e-4. Validation losses are synthetic-task results, not language-model benchmarks. The source work/work/recurrent_ctx_interaction_v4_causal.py is included in the local work tree.
| Configuration | Params | Validation loss | PPL | Train seconds |
|---|---|---|---|---|
| recurrent, context 64 | 1,397,504 | 1.80549 | 6.0830 | 10.45 |
| recurrent, context 256 | 1,446,656 | 2.23559 | 9.3520 | 25.86 |
| stacked, context 64 | 1,402,880 | 1.92167 | 6.8324 | 7.93 |
| stacked, context 256 | 1,452,032 | 2.27253 | 9.7039 | 13.69 |
Result
Moving from context 64 to 256 raised validation loss by 0.43011 for the recurrent model and 0.35086 for the stacked model; longer context did not help under this training setup. Recurrent models had lower loss than stacked counterparts at both lengths (advantages 0.11618 at 64 and 0.03693 at 256, computed from the recorded losses). This does not show a benefit from longer context or an amplified context benefit. A single run per condition, synthetic data, and the modest 2,000-step budget limit conclusions; seed replication and task/data inspection would be needed before generalizing.
Reproduction
From a checkout containing the source file, run python3 -u recurrent_ctx_interaction_v4_causal.py. The recorded exact results are in recurrent_ctx_interaction_results.json; source and JSON are preserved alongside this report in this sandbox. The original run used CUDA, as recorded in its result configuration.