ngram embeds | DDL new code + EC | Hyperball (AdamH, AdEMAMixH) | lr_expert

SLW end

DDL

STEM

fix n_kv_heads + pos_id GTPAv2

M3 MIMO large output fix (see discussion)

FA3 fixes | some refactoring

equivalence with MG (updated GTDPA, GDN, M3MIMO ) | some refactoring

Grouped Differential Attn v2 | GDAv2 for TPA

GDN proj same as in MG

CDP LR expert scaling tests (linear or sqrt, inc) | missing args in config (not critical) |

Differential Attn v2 | proper coord check for MoE | proper init for MoE | some refactoring

attn layer simplification & IDM (for gpt baseline) | local/global removed | CDP coord check | CDP fix lm_head init scaling |

vwn start

coordcheck v2 (gqa paper) | some fixes relative to previous rehaul

big overhaul (simplication, removal of unused things)

some prints and imports fixes

CompletedP | memory+norm logging | proper MoE with ScatterMoE, update bias, Latent-MoE | Muon experiments | VE for Mamba3 | fix torch recompiles during varlen training

Value Embeddings (attn & gdn) | MoE (dont use this one)

alpha normalize ademamix | mamba norms and gate | VWN | wnorm (nemotron-flash) | MG equivalence | fix IDM config saving | CCAv2 | MoBA | reduce lm head

head tying | gated mlp | gate of Mamba3 inside module

mamba3 flags | mamba3 default state size to 128, headdim to 64 | mamba2 | fix mamba3 mimo (JG) | (fake) moe | intra doc maskiiiing (with SS) | seednorm tests | coord checks

MLA | KDA | TPA | GDA | ResFormer | Mamba3 | DragonMimo (WIP) | tokenshift | SeeDNorm | shrink DA/GDN | gate shared across all block types |

CCE | Gate attn | ZCG | RoPE GDN | GQA GDN | uniconv GDN ||CCA | NSA | PLT (not tested) | DMA fix | SWR

Update training_dragon.py

fix massi bugs prod

three conv (for now), works with main MG branch

uscaling training | resume training | slw training | eval loss training | fix data offset training | DSA & DMA (testing) | Qwen3Next-like arch

fixes+refactoring

zero centered gamma for norm | proper qkv proj in GDN (tp aware, same as MG) | training script (wip)

revamp GDN cache (as QwenNext) & conv1d

refactor backends selection | fix eager attn softcap & window | fix flex backend window | flex attn & eager backends for DA | eager backend for GDN | refactor GDN variables

flex attn backend for ATTN (tested) [ inc

revert vLLM modifications (separate head mult back to lm_head)

merged three convolutions of GDN (test on PIQA&SWDE)

merged in_proj and gate_proj of GDN (tested on PIQA)

Update configuration_dragon.py

max pos embeddings

manual automap for DragonModel | vLLM compat (alpha head in model, persistant=False, contiguous for conv1d, max pos hardcoded for now)

diff attn backend FA works (eager, no)

diff attn FA2+FA3+eager backends (WIP)

removed loss reduction mean

added num_hidden_layers in config for generate function

old LNS variant to be compatible with old MG (<uscaling)

nothing

updated rope (uses position_ids) | updated scalable softmax (uses position_ids) | refactoring | gitignore

relative import

correct kv cache management (trim after) | register for auto class

slw size in config | updated cache with SLW | fix gdn scalar (still needs refactoring)

swa (for local and diff)