Mamba-3 vs Transformer vs Hybrid - Bangla case study checkpoints (5 seeds)
Three ~24.5M non-embedding-parameter causal LMs trained from scratch on 1B tokens of
Bangla (FineWeb-2 ben_Beng), 5 seeds each (1337, 2024, 3419, 5150, 8888):
- Llama-style Transformer (RoPE, SwiGLU, full attention)
- official Mamba-3 tower (state-spaces/mamba)
- hybrid: Mamba-3 backbone with attention at 2 of 15 layers
Findings (5 seeds): (1) Mamba-3 and the hybrid beat the Transformer on perplexity (all seeds). (2) The Transformer's subject-verb agreement degrades with subject-verb distance in every seed; Mamba-3's does not (the hybrid degrades like attention). (3) Most between-architecture agreement gaps (incl. honorific) are NOT significant across seeds - Mamba-3 is far more seed-sensitive than the Transformer.
results/ holds per-seed perplexity + per-condition probe accuracy (and seed-1
item-level per_pair/ CSVs used for the McNemar contrast). checkpoint.pt is
weights-only; load with bangla_ssm.models.build_model and the matching config.json.
Code + probes: https://github.com/sahilaf/Mamba3_Bangla_Case_Study Probe dataset: https://huggingface.co/datasets/sahilfarib/bangla-agreement-probes