How is this different from Qwen3.8-Flash-Next?

#3
by theJuice85 - opened

This looks like a repack what specifically is done differently in the SFT, quantization, or architecture?

Not a repack — we rebuilt the expert layer by changing moe layer into latent moe for on-device inference and then post-trained the model. The architecture of model has been totally changed

Qwen3.8-Flash-Next carries 120.8 B parameters of routed experts. Edge0 makes that 84.6 B on the 145B (−30%) and 60.4 B on the 120B (−50%). On disk the expert pool goes from 46.84 GiB to 37.32 GiB and 26.66 GiB, and the parameters each token pushes through the experts drop from 2.36 B to 1.65 B and 1.18 B — that is the number that sets decode speed and the memory a device needs.

It is not a conversion of Qwen's tensors: after the architecture change we post-trained the model for a long schedule on our own data, and it is trained natively at 4 bits rather than quantized afterwards. Quality holds — the 145B is ahead on the agentic benchmarks (deep-swe 64.6 vs 58.7, terminal-bench 86.5 vs 83.3) and level elsewhere.

On one RTX 5090 D with 16 GiB VRAM and 20 GiB RAM, decode is 2–3× Qwen's: 18.5 tok/s for Qwen3.8-Flash-Next, 62.4 for Edge0-120B and 37.7 for Edge0-145B.

Ok so expert pruning + shared expert healing + SFT on local trajectories (training dataset available?) to boost specific benchmarks (termbench2.0 or 2.1 or 4.0?). Any work on the MTP head or PLE table? Any distillation?

Edge0 org

Ok so expert pruning + shared expert healing + SFT on local trajectories (training dataset available?) to boost specific benchmarks (termbench2.0 or 2.1 or 4.0?). Any work on the MTP head or PLE table? Any distillation?

We'll be publishing a technical report soon — all the details, including methodology, ablations, and benchmarks, will be covered there. Stay tuned

SO, the official benchamark in made using the nvfp4 checkpoint(https://huggingface.co/Edge0/Edge0-145B-NVFP4), is it?
AND the dtypes for decoding?
OR which checkpoint is better for the quality, with enough gpu?

Edge0 org

SO, the official benchamark in made using the nvfp4 checkpoint(https://huggingface.co/Edge0/Edge0-145B-NVFP4), is it?
AND the dtypes for decoding?
OR which checkpoint is better for the quality, with enough gpu?

“Edge0-145B” refers to the NVFP4 model—it is the exact checkpoint we released. It was trained natively in NVFP4 rather than quantized afterwards, so there is no higher-precision version; the benchmark numbers are from the released weights.

wanglamao changed discussion status to closed

Sign up or log in to comment