inclusionAI

/

Ling-3.1-flash

Like 157

Follow

inclusionAI 3.42k

I recently converted and ran Ling-3.1-flash locally and wanted to share a few implementation details that were easy to miss, in case they help others. The model itself has been solid once these were handled; the notes are about the reference code path rather than the weights.

1. The SwiGLU limit lists are load-bearing, but the HF reference does not apply them
   config.json sets share_expert_swiglu_limit_list to 10 for layers 64–71 and expert_swiglu_limit_list to 10 for layers 65–71. SGLang's BailingMLP applies min(silu(gate), L) * clamp(up, -L, L), but modeling_bailing_moe_v3.py ignores both lists. Without the shared-expert clamp, the layer-64 output jumps by roughly 5x and the continuation becomes incoherent; with it, the model immediately starts reasoning normally. It would be worth either implementing the clamps in the HF reference or documenting that HF-logits and SGLang-logits intentionally differ on those layers.

2. Every routed expert has its own global scales
   The per-expert weight_global_scale values are not shared per layer. If a loader probes one expert and reuses its scales for the rest of the layer (a tempting shortcut for a 512-expert layer), the model still loads without errors but produces fluent-looking garbage. Reading each expert's own scales fixed it.

3. The tokenizer is not the o200k/Llama-3 pre-tokenizer
   The pre-tokenizer is a single letter run with an optional prefix, one digit per number word, and apostrophe contractions:
   '(?i:[sdmt]|ll|ve|re) | [^\r\n\p{L}\p{N}]?+ \p{L}+ | \p{N} | ...
   Using the common Llama-3/o200k case-run split changes the token ids (getUserName is one pre-token, not three). Worth pinning explicitly if you compare token streams.

4. Architecture details that matter for caching

• 72 main layers + MTP layer 72; the MTP layer is a full MLA + MoE layer and is not part of the normal forward pass.
• MLA is on layers 7, 15, …, 71 (every 8th layer), with a head-wise sigmoid output gate.
• KDA has direct f_proj / g_proj projections (no low-rank split), conv-4, kda_safe_gate, lower_bound = -5, and per-head L2 normalization with eps = 1e-6, then q *= 1/sqrt(128).
• MLA uses 1/sqrt(192) attention scaling, interleaved RoPE on 64 of the 192 qk dims, and no YaRN.
• Layers 0–3 are dense MLP; 68 MoE layers follow.

5. Late-layer activations are very large
   The residual stream reaches ~1e5 squared-norm by the last layers. That is fine in BF16/FP16, but it makes the model sensitive to 4-bit quantization of the dense path and of the shared expert. In my NVFP4 conversion, a 4-bit-only body was enough to flip near-ties in the first generated token; keeping the dense/attention path at higher effective precision was what made the output stable. If someone is quantizing this model aggressively, I would check the first 1–2 generated tokens against the source before trusting a long run.

6. A single-RTX-5090 data point (not a benchmark)
   With a 4-bit body + streamed 4-bit experts, the dense side is ~10.2 GiB pinned and the ~287 GiB expert set is mmap-streamed through an LRU cache (16–17 GiB on this card). At 2048 generated tokens I measured 2.8 tok/s (greedy, auto fast-math) and 2.4 tok/s in a bit-exact mode, with 10 s TTFT. The control prompt produces a normal reasoning trace and answers “Paris”. This is obviously far from the intended 8×H200 setup, but it is usable for correctness checks.

Minor observations

• max_window_layers and use_qk_norm do not appear to affect the HF forward path; linear_silu, kda_safe_gate and the two SwiGLU limit lists do.
• The chat template defaults to detailed thinking on, and the assistant turn ends with ASSISTANT\n; the tool format is the name…… XML.

Thanks to the authors for releasing the weights and the reference code. If the missing SwiGLU clamps in the HF file are intentional (i.e. the HF path is meant to match a pre-clamp checkpoint behavior), a short note in the model card would save the next person a day of debugging.