TMT: dynamic graph attention beats Mamba on WikiText-2 at 48% compute — open source
#34
by vigneshwar234 - opened
TemporalMesh Transformer — benchmarked on this dataset
TMT achieves 29.4 PPL on WikiText-2 (vs 31.8 Mamba, 42.1 vanilla) and 36.1 on WikiText-103 at only 48% relative compute — 120M params, 3 seeds.
Five innovations: Mesh Attention (O(S·k) dynamic kNN), Temporal Decay, Adaptive Exit Gate, Dual-Stream FFN, EMA Memory Anchors.
📄 Paper: https://zenodo.org/records/20287390
💻 Code: https://github.com/vignesh2027/TemporalMesh-Transformer
🎮 Demo: https://huggingface.co/spaces/vigneshwar234/TemporalMesh-Transformer-Demo