analysis/context_length_single_seed.md

Recurrent depth × context length: a bounded synthetic comparison

Question

Does recurrent depth improve the benefit of longer context at tiny scale?

Setup

This report summarizes the completed recurrent_ctx_interaction_v4_causal run (exit 0; 63.2 seconds), recorded in recurrent_ctx_interaction_results.json. Four configurations were trained for 2,000 steps, with d_model=256, 4 heads, byte vocabulary 256, batch size 32 and learning rate 3e-4. Validation losses are synthetic-task results, not language-model benchmarks. The source work/work/recurrent_ctx_interaction_v4_causal.py is included in the local work tree.

Configuration	Params	Validation loss	PPL	Train seconds
recurrent, context 64	1,397,504	1.80549	6.0830	10.45
recurrent, context 256	1,446,656	2.23559	9.3520	25.86
stacked, context 64	1,402,880	1.92167	6.8324	7.93
stacked, context 256	1,452,032	2.27253	9.7039	13.69

Result

Moving from context 64 to 256 raised validation loss by 0.43011 for the recurrent model and 0.35086 for the stacked model; longer context did not help under this training setup. Recurrent models had lower loss than stacked counterparts at both lengths (advantages 0.11618 at 64 and 0.03693 at 256, computed from the recorded losses). This does not show a benefit from longer context or an amplified context benefit. A single run per condition, synthetic data, and the modest 2,000-step budget limit conclusions; seed replication and task/data inspection would be needed before generalizing.

Reproduction

From a checkout containing the source file, run python3 -u recurrent_ctx_interaction_v4_causal.py. The recorded exact results are in recurrent_ctx_interaction_results.json; source and JSON are preserved alongside this report in this sandbox. The original run used CUDA, as recorded in its result configuration.