Horizon bottleneck at 128-byte sequence length and associative state updates
Hi team,
Scaling a continuous-time / Liquid Neural Network (MT-LNN) architecture to 1.93B parameters on a single A100 is a fascinating research milestone, and 1.45 BPB on WikiText-103 shows strong local byte-level predictive capability.
Regarding the "honest-null on reasoning mechanisms" finding, the limitation appears directly tied to the temporal horizon rather than model capacity:
Temporal span of byte-level tokens at seq_len 128:
With a 256 UTF-8 byte vocabulary, average English word fertility is roughly 4.5 to 5 bytes per word. A sequence length of 128 bytes limits the entire operational context to ~25-28 words. Multi-step reasoning chains or premise-hypothesis deductions require at least several hundred words (1,500 - 4,000 bytes) to formulate a relational context. At 128 bytes, the model cannot hold both the problem premise and resolution steps within its active horizon.State stability across 34 continuous layers:
Scaling sequence length in deep (34-layer) continuous-time ODE / liquid formulations typically triggers severe gradient degradation or simulation slowdowns during BPTT on hardware accelerators.
In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we handle long-horizon sequence modeling using Gated DeltaNet (GDN-2) linear recurrence:
Instead of numerical differential equation solvers, GDN-2 maintains an O(1) associative memory matrix updated via closed-form outer products and data-dependent decay gates.
Because the recurrence formulation maps directly to hardware-efficient chunkwise parallel scans during training, it allows byte-level sequence models to extend context to 8,192+ bytes without vanishing gradients or throughput collapse.
Pairing this linear recurrence with sparse global attention layers at a 3:1 stride restores long-range reasoning recall while preserving O(1) constant-memory decoding.
Did numerical stiffness or BPTT gradient instability in the MT-LNN state updates enforce the 128-byte context boundary during this 30k-step run?
Best,
Andrew
Hi Andrew,
Thanks for the careful read - and for the Maba reference. GDN-2's chunkwise-scan formulation is a clean way to push byte-level context past 1K, and pairing it with sparse global attention at 3:1 is a pragmatic move to keep recall while preserving O(1) decoding.
To your direct question: no - neither numerical stiffness nor BPTT gradient instability enforced the 128-byte boundary. seq 128 was a training-budget configuration, not a stability limit:
- This was the first complete 30K-step run of the 1.93B preset (2080x34x16) on a single A100-PCIE-40GB - bf16 model, 8-bit optimizer, lr 1e-4 (set conservatively for optimization stability at this scale). Activation memory for BPTT through the full 34-layer stack was the binding constraint; the run even survived OOM interruptions from GPU contention, recovered through checkpoint discipline. No state-update divergence was observed.
- We have since extended the same run to 60K steps (val PPL 2.73 -> 2.556 from 30K to 60K - still improving, not saturated).
The most relevant evidence for your question: we evaluated the 30K checkpoint at longer sequences with no retraining. PPL is flat to 4x the training length: 2.72 @128 / 2.72 @256 / 2.68 @512. The O(1) recurrent state extrapolates beyond the trained window rather than collapsing - so 128 bytes was a budget choice, not an architectural horizon of the state update.
Where your framing is right is the reasoning side - though the blocker we measured is a different one: our depth/iteration probes on pointer-chase-style tasks (public in the M1 repo) point to the learnability of iterative composition under SGD at these budgets, rather than the state's ability to carry information. But we fully agree that ~25-28 words of active context is far too short for premise-hypothesis reasoning, and extending the trained context is on our roadmap - the flat-PPL result above is our evidence that the architecture itself is not the blocker.
On the write-rule side: we have been benchmarking memory-write mechanisms in an associative-state setting - delta-rule family vs gated/regression writes, capacity walls vs interference, and where evidence-weighted gating breaks in open-world binding streams. Quite adjacent to GDN-2's territory; happy to compare notes or share benchmark details if useful.
Thanks again - this kind of review is exactly why we published the checkpoint with its boundaries stated.
Best,
Everest - AwareLiquid
Hi Everest,
That zero-shot sequence length extrapolation is a really clean validation. Seeing PPL hold flat to 512 bytes (2.72 down to 2.68) confirms the continuous-time update operator is genuinely contractive rather than drifting into numerical runaway outside the training horizon.
Regarding activation memory capping BPTT at 128 on a 40GB card:
That is exactly where gradient checkpointing bounded per physical block or chunkwise scan reformulations become essential. When the backward pass only needs to retain boundary states between chunks rather than all intermediate activation tensors across all 34 layers, context can scale by an order of magnitude on the exact same A100.
Your observation about pointer chasing and the learnability of iterative composition under SGD hits the core bottleneck of recurrent models:
Single-pass continuous states struggle with multi-hop pointer dereferencing because SGD has trouble discovering clean iterative addressing without explicit non-linear depth. This is precisely why in Maba we test deterministic 2-pass recycling: routing through the physical state blocks a second time gives SGD the compositional stages needed for multi-step pointer resolution without adding physical parameters.
Comparing notes on memory-write mechanisms would be great. In associative state tracking, standard delta updates handle interference well via orthogonal projections, but data-dependent decay gates often struggle when fast-changing binding pairs overwrite long-term background constraints.
Would be very interested to see your benchmark details on the capacity walls across the delta versus regression write variants. What is the best place to swap notes, here or via email?
Best,
Andrew