Harmonic dissonance across multi-track voices and context scaling past 35 bars
Hi Haster,
Training a 0.2B symbolic music model from scratch in fp8 on a Blackwell RTX PRO 6000 with native bar-major REMI+ and unpitched GM drum resolution is a fantastic release. Interleaving tracks bar-by-bar to enforce co-termination is very clever engineering.
Looking at your evaluation table and the limitations noted regarding the 35-bar context ceiling:
Long-horizon thematic continuity versus CONTINUE window stitching:
Because bar-major interleaving packs multiple instrument events into every bar, a 4,096-token window spans only roughly 35 bars of dense musical score. Stitching pieces via CONTINUE windows causes harmonic and stylistic drift at the seams, as previous motifs drop entirely out of attention.
In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we handle extended sequence memory using hybrid linear recurrence (75% GDN-2 / 25% GQA):
GDN-2 maintains an associative recurrent state in fixed O(1) memory.
In symbolic music, this allows the network to carry harmonic keys, motif seeds, and rhythmic themes across hundreds of bars without quadratic memory growth or window-stitching boundaries.Polyphonic voice leading and dissonance reduction via block recycling:
Your dissonance ratio landing at 0.29 (compared to 0.15 on human reference scores) and in-key ratio at 0.81 reflect the difficulty of multi-voice constraint satisfaction across 17 layers. Resolving vertical intervals simultaneously across lead, chords, and bass within each bar requires deep non-linear feature interaction.
Using deterministic 2-pass block recycling:
Passing representations through your 17 physical blocks twice with Split RMSNorm (distinct scale vectors for pass 0 and pass 1) expands depth to 34 effective layers at zero parameter overhead.
Pass 0 establishes local rhythmic and track alignment, while pass 1 enforces strict harmonic intervals, directly reducing clashing notes and dissonance across simultaneous tracks.REMI+ vocabulary parameter reallocation:
With a 9,507 vocabulary at 1,024 hidden dimension, your embedding table takes 9.73M parameters.
At ~11.3M parameters per transformer layer, decoupling the input vocabulary via low-rank factorization (9,507 -> 128 -> 1,024 = 2.44M params) reclaims ~7.3M weights, covering most of the budget needed for an 18th physical layer within your 201M envelope.
Did memory bandwidth or fp8 kernel scaling dictate stopping at 17 layers during the Blackwell run?
Best,
Andrew
Hi Andrew,
Really appreciate the detailed feedback—honestly one of the best reviews I’ve gotten on this, especially since you actually dug into the eval table and codec notes.
To answer your question on the 17-layer / d1024 setup: it wasn't constrained by memory bandwidth or fp8 kernels. I just carried the architecture over from an earlier bar-major 200M run to keep the param budget consistent. The Blackwell fp8 path actually had plenty of headroom left, so the depth was purely a budget choice, not a hardware bottleneck.
You're spot on about the long-context issue. With bar-major interleaving, a 4k token window only covers ~35 dense bars, and CONTINUE-stitching definitely starts drifting once early motifs fall out of view. A hybrid linear-recurrence setup (SSM/GDN for O(1) state + a few attention layers for recall) is definitely the right fix, and Maba seems like the right direction here.
My main blocker is just compute. I'm training on a single Blackwell card tuned for fp8 to keep costs manageable, but in my experience, selective scan / linear recurrence kernels pretty much require bf16 to stay stable. Dropping fp8 tanks the throughput to where it's just not viable on my current setup—I’d realistically need a few weeks of dedicated H100 time to train a hybrid from scratch. It’s definitely on my roadmap, but I’ll probably try cheaper options first (like extending context length during training or inference-time RoPE scaling) before rewriting the architecture.
The 2-pass block recycling with split RMSNorm is an interesting idea—hadn't thought about depth recurrence quite that way. My only hesitation is the 2x FLOP hit for the same parameter count (which runs into the same compute bottleneck), but it's cheap enough to prototype that I might give it a spin. On embedding factorization: fair point, but my embeddings are already tied (input = output). Factoring it out only frees up ~7–8M params, which doesn't even buy a full ~11M-param layer at a 9.5k vocab, so the depth vs. embedding capacity trade feels pretty much like a wash. Still a neat micro-optimization to keep in mind.
Thanks again for the thoughtful thoughts—I'll definitely give Maba a closer look.
Best,
Haster
Hi Haster,
Really glad the review was helpful!
On the fp8 stability issue with linear recurrence: that used to be a real blocker with token-by-token scans, but chunked formulations make a huge difference. All the heavy projections (Q, K, V, gates) run as normal fp8 GEMMs at full tensor core speed, and only the chunk-level state accumulation needs an fp32 buffer to stay stable. Even in pure PyTorch with torch.compile, keeping just the recurrent state update in fp32 while the rest runs in fp8 avoids gradient explosions without tanking throughput.
As for the 2x FLOP hit with block recycling, you are totally right that it doubles compute per token. The upside we observed is better sample efficiency: the added depth drops loss faster per token seen, which partially offsets the per-step compute cost. But on a single card with strict wall-clock limits, testing RoPE scaling or a context bump first is definitely the pragmatic way to go.
Also completely agree on your tied 9.5k vocab, at that scale the savings are pretty marginal compared to models dragging around 32k or 128k tables.
Really cool project with Clef, looking forward to seeing where you take it next!
Best,
Andrew